Bro 2.5 and log rotation

Zeek
1

I guess I’m in this boat as well. Since my upgrade, bro will stop rotating logs at some point. I’m not running bro via broctl. Here’s my process for log rotation:

local.bro:
redef Log::default_rotation_interval = 86400 secs;
redef Log::default_rotation_postprocessor_cmd = “archive-log”;

broctl.cfg:
LogRotationInterval = 86400

sudo /usr/local/bro/bin/broctl install

sudo ln -s /usr/local/bro/share/broctl/scripts/archive-log /usr/local/bin/
sudo ln -s /usr/local/bro/share/broctl/scripts/broctl-config.sh /usr/local/bin/
sudo ln -s /usr/local/bro/share/broctl/scripts/make-archive-name /usr/local/bin/
sudo ln -s /usr/local/bro/share/broctl/scripts/expire-logs /usr/local/bin/
sudo ln -s /usr/local/bro/share/broctl/scripts/delete-log /usr/local/bin/
sudo ln -s /usr/local/bro/share/broctl/scripts/cflow-stats /usr/local/bin/
sudo ln -s /usr/local/bro/share/broctl/scripts/stats-to-csv /usr/local/bin/

This will work for a while. But at some point it stops:

Screenshot from 2016-12-22 05-58-45.png1622×200 50.8 KB

at the core I believe it’s because bro, after sometime, won’t respond to a “normal” kill command. A “sudo killall bro” will do nothing. Usually I’ll “sudo killall bro”, wait a minute, and then my spool directory will be empty, I’ll have an email with stats, and I’ll have my new archive directory. I’ll have to -9 it in order to get it to stop, I’ve restarted this morning and will see how many days it will go. Thank you.

James

2

I've seen this before when people are generating really huge logs and IO on their system goes crazy because the previous logs are still being compressed which runs into a downward spiral that it never recovers from. For those logs that you have which haven't been rotated as you expected, was there a gzip process running in the background? I suspect that you have a lot of gzip processes running and a very high system load.

  .Seth

3

Thanks Seth,

Interestingly, this is on my home network…the largest compressed file in looking at past logs was tcprecovery at 7.8 megs. On a hunch, after this issue came up again on Christmas day, I disabled TCPRS and have had no issues since.

James