Bro 2.5 Logger crash --> Broken Log Directory naming

Zeek
1

Hello all,

I’ve recently come to be responsible for a Bro server and am doing my best to keep everything running smoothly at the moment.

We are running a cluster configuration on a single physical machine. Recently we updated to Bro 2.5 from 2.4. Additionally, we modified our cluster configuration to enable a 1 logger process alongside 16 workers, 3 proxies, and 1 manager process (prior we were running without the logger and were seeing the manager crashing regularly due to memory constraints).

The output log file structure has had a strange file naming for a short period of time around 2am last night. It seems that the incorrect file naming may correspond to the logger having crashed. It seems when the Logger process is being brought back online by the Broctl Cron task, the logger logs to a strange directory naming for some short period of time.

Strange log directory naming:

[/bro/logs]$ du -h 20*

1.7G 2000-00-

1.7G 2000-59-

3.3G 2010-00-

67G 2016-12-21

160G 2016-12-22

84G 2016-12-23

1.9G 2020-00-

1.6G 2021-16-

5.1G 2030-00-

8.0K 2030-16-

3.2G 2040-00-

1.9G 2040-10-

1.7G 2050-00-

1.9G 2050-05-

How can I ensure that when the logger comes online after a crash that it won’t use a strange directory naming?

Thanks for any thoughts or help!

Best Regards,

-Ryan

LoggerCrashReport.txt (4.09 KB)

2

When our cluster becomes unstable we see the same behavior. I think making the cluster stable is the answer. I suspect you have the same problem we do in that the logs are not written to disk fast enough and slowly buffer all memory until Bro crashes.

I think the answer is to enable multiple loggers and Kafka export but I’ve not figured out how to do that yet.