log delays and logger CPUs

I have a cluster running Bro 2.6.4. One host runs a manager and logger, 8 other hosts run proxy and worker nodes.

Lately the logger node has not been able to keep up with the logs, and I’ve noticed that the most recent entries in the current/conn.log are significantly delayed (I’ve seen delays as high as 90 minutes).

The logger process has maxed out CPU usage on core 1. The node.cfg file specifies 8 CPU cores (all on the same NUMA node as the NVMe drive where the logs are written):

[logger]
type=logger
host=bromanager-01.umn.edu

pin_cpus=1,3,5,7,9,11,13,15

broctl nodes shows that only 1 CPU core is pinned:

/usr/local/bro/bin/broctl nodes
logger - addr=10.x.x.x aux_scripts= brobase= count=1 env_vars= ether= host=bromanager-01.umn.edu interface= lb_interfaces= lb_method= lb_procs= name=logger pin_cpus=1 test_mykey= type=logger zone_id=

Can pin_cpus be used with a logger node? Any other suggestions for improving logger performance?

Ah… pin_cpus is more intended to work with multiple worker processes. It’s definitely doing the wrong thing in your case. It’s pinning the first logger (out of one total) to core 1, and then it never uses 3,5,7… You’re better off removing that setting. It should run fine across all cores, even with the numa hit… the volume of logs that would go across the numa bus would only be a small fraction of the total bandwidth.

You could still pin it to those cores, but you’d have to do it manually using taskset for now. This is probably something that could be fixed in broctl(zeekctl) to better handle pin_cpus option when only a single process is being started.

So there’s a couple of things you could do.

  1. Indeed using pin_cpus for the manager or the logging will result in all threads being pinned to the same CPU, definitely not ideal and worth filling a bug

  2. You could (per Justin’s idea from Zeek Week) use the path func and write a script that will split biggest log files into two or more files - and each will get a separate thread

https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_dns.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_http.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_intel.zeek

  1. And maybe implement some filtering for traffic you don’t care for, if there is any, examples here

https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_files.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_mysql.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_ssl.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_x509.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_conn.zeek
https://github.com/michalpurzynski/bro-gramming/blob/master/filter_input.zeek