Bro + Log rotation (solr ?)

Zeek
1

Hey all,

I’m using Bro + rsyslog filereader in order to pump Bro into our big data solution (Apache SOLR). I’m using custom python scripts to parse the incoming bro messages, batch them into appropriate sizes, and then POST them to the SOLR cluster we have setup. The main problem i’m running into is that rsyslog does not seem to ‘follow’ the files once they have gone through a Bro logrotate. Is there a way to completely disable logrotate? Has anyone had any luck with the Bro logrotate and not ‘losing’ file handles?

I’d love some help in this matter. Also - i know that Bro supports elastic search POSTing (via libcurl). Is there any reason why an apache SOLR module can’t be written/adapted? I don’t see a need to write to a file and worry about file handles, when it’s almost exactly the same to POST to SOLR as it is to ES. Since it’s all libcurl (and JSON) under the hood, i’d be glad to post/share the SOLR schemas i’ve created for the Bro data.

Thank in advance.

Cheers,

JB

2

I experienced the same thing, but since I rotate the files manually, I restart the syslog service after rotating and that's done the trick for me.

James

3

Hey Joe,

It is possible with the current setup to write your own logging utility to pipe events directly to your system of choice. Since SOLR is REST-based just copy over the ElasticSearch module and do some code tweaking.

Be aware that the devs are working on a new modular method for extending Bro that will include logging. Should hopefully be a less-painful migration.

4

Hey James,

How exactly are you completely disabling the bro file rotation? This is why i tried in broctl.conf:

SitePolicyStandalone = local.bro
CfgDir = /usr/local/bro/etc
SpoolDir = /usr/local/bro/spool
LogDir = /usr/local/bro/logs
LogRotationInterval = 0
MinDiskSpace = 5

But i still see gz files being created. Am i missing something to completely disable?

Cheers,

Justin

5

I don't use broctl, I use bro command line only. Something like:

/usr/local/bro/bin/bro --no-checksums -i eth0 local "Site::local_nets += { 192.168.1.0/24 }"

James

6

After changing broctl.cfg, did you remember to run
"broctl install"? Your changes do not take effect until
you "install" them.

Next, you need to restart Bro ("broctl restart") so that
Bro will read the new settings.