logging locally and to remote logger

Zeek
1

So, if I use:

redef Log::enable_local_logging

in a bro worker cluster, what I find is that all the logs go to /data/bro/spool/worker-1-X instead of all in /data/bro/logs/current on the local machine… Is there a way to fix this?

Also, I would want to rotate logs out on the workers that are doing additional local logging to have a much more constrained timeframe for logging, specifically 1 week for local nodes, and 3 months for the logger host.

Is the best way to do this just with a cron rm -rf /data/bro/logs/$date ? It seems this would run into a conflict with broctlconfig…

Thanks!

2

Hi,

If you wish to log locally and you care about the worker-id who produce this logged event :

  • to know what is the worker-id you can add a field “worker” to your logs and populate it from bro script using : get_event_peer()$descr

  • to change the rotation for each log (here, rotate every 200 minutes) you need to use

  • LOG::remove_default_filter(SSH::LOG);

  • and then add LOG::add_filter(SSH::LOG, [$name=“ssh”,$path=“ssh”,$interv=200min, $include=(“field1”,“field2”) ]

  • btw, you can set $path to be a mounted dir- to save the log to another machine simultaneously :

  • use bro, add a new writer (https://www.bro.org/sphinx/scripts/base/frameworks/logging/main.bro.html#id-Log::default_writer) and then add_filter to ssh and ask it to use the new writer

  • use syslog, just monitor this main local log and transmit it to another machine
    Hope it helps

3

Ah, I think there is some confusion. Out of the box if you log locally as well as using a remote logger (2.5), the logs locally get shoved into worker buckets. I was hoping to see how it would be possible to get standard cluster behavior, where all workers log locally to one bucket instead of each worker having its own bucket.

Anyone know why this logs to separate buckets in the first place?

4

Hi Erik,

the workers log to one directory each, to not conflict with each other. If
you have several active workers on one machine, they cannot local-log to
the same directory/file because they would conflict with each other and
you would get files where different workers might write into lines of
other workers.

As soon as you want merged logs from more than one Bro instance, you need
remote-logging (even if the manager/logger for the workers is on the same
machine).

I hope this helps :slight_smile:
Johanna