Feature Request, up to 50% done?

Zeek
1

Hi,

When I need to search my bro log files, I find myself using ‘zfgrep’ quite a bit…Out of the box this works fine, however I lose the ability to translate epoch time to ‘human readable’ time. The examples I’ve seen suggest “zcat $file | bro-cut -d” however this means that every line must be run through awk, which greatly increases search times. for an example search of a one hour http file, my ‘bcut’ method runs in 6.9 seconds, while using bro-cut properly(? “zcat file | bro-cut -d | fgrep string”) takes 5 minutes, 1.2 seconds. Doing my zfgrep before the bro-cut causes only blank lines to be printed (I assume because there are no description fields to be read).

The workaround I’ve been implementing is to:

alias bcut

alias bcut=‘awk -f /usr/local/bin/epoch_to_human.gawk’

cat /usr/local/bin/epoch_to_human.gawk

#!/bin/gawk
{
val=strftime("%Y-%m-%dT%H:%M:%S%z", $1, 0)
$1=val
print $0
}

zfgrep ‘10.10.10.10’ /usr/local/bro/logs/some_log.txt.gz | bcut

This works well enough. Tt would be nice if there were a switch to bro-cut that would implement this functionality, however I’m unsure of how to integrate it myself (most of my awk programs are one line throwaways). The key would be make it clear that you cannot specify field selections with bro-cut, that this would only attempt to translate the first field into a “human readable” format. Some error checking is likely in order as well…

So, I suppose I’m requesting that someone with more gawk chops than myself give a shot at integrating this into bro-cut, or give me some pointers and I might be able to stumble through creating a patch for submission…Or, perhaps I’m approaching this problem in the wrong way, and could use a pointer on a better way to go about it (aside from “ship the logs elsewhere that indexes searches”; we’ll deal with that scenario later) :slight_smile:

Cheers,

Jesse

2

I tend to use these lines in my profile...

alias bro-column="sed \"s/fields.//;s/types.//\" | column -s $'\t' -t"
alias bro-awk='awk -F" "'
bro-grep() { grep -E "(^#)|$1" $2; }
bro-zgrep() { zgrep -E "(^#)|$1" $2; }

What you're trying to do can then be accomplished like this…

bro-zgrep '10.10.10.10' /usr/local/bro/logs/conn.*.log.gz | bro-cut id.orig_h,id.resp_h

It *would* be handy to be able to do this through bro-cut though but that would make bro-cut start to sound like an incorrectly named utility. :slight_smile:

Have you tried using the ElasticSearch writer and Brownian?

  .Seth