Hi,
has anybody already written a python script or module to easily parse the new Bro ASCII log files?
cu
Gregor
I've got some code for this, but it's a work in progress; you're welcome to take a look at what's there, though, and do whatever you'd like with it.
*** Note that this code demands a slightly modified ASCII header (which includes the bro path name and the separator char).
Anyway, branch is:
topic/gilbert/log-util
and check: aux/log-util
I think the usage is pretty straightforward:
manager = BroLogManager()
manager.load('/path/to/logs')
#If this is a directory, *all files from all subdirectories* of this path will be loaded.
# Otherwise, only the referenced files will be loaded.
# .log, .log.gz, and .log.bz2 are treated as new-style bro logs
print manager['conn'].get_stats('orig_bytes')
# This prints some interesting statistics about the orig_bytes column (min / max / mean / std_dev).
# Note that the above will actually load *all* relevant log files and perform the calculation, then cache
# the results; the first get_stats() will be very slow, but the rest should work pretty quickly.
for e in manager['conn'].entries():
print e['ts'] # Note that 'e.ts' will also work, but that this doesn't work for field names that don't map nicely to Python.
See 'bro-logtool' for a toy script I've been using to play with the library.
--Gilbert