Distributed deployment high cpu low network traffic

Zeek
1

We’re getting around 30-160 mbps worth of traffic on our sensor and when we use multiple workers they are all pegged 96% to 99% cpu utilization. I’ve tried disabling our custom scripts and that wasn’t the culprit. I’ve tried 4 workers and 8 workers, I’ve tried pinning them to different CPUs and nothing has changed anything. We’re using PF_RING. Any idea how I can trouble shoot this issue? We’re having packet loss, but with the low speed of traffic I wouldn’t expect to see any loss.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32027 root 20 0 1527600 1.430g 11696 R 99.9 4.7 10:37.82 bro
32018 root 20 0 1520192 1.423g 11792 R 99.6 4.6 10:38.83 bro
32019 root 20 0 1525308 1.428g 11780 R 99.6 4.7 10:39.06 bro
32032 root 20 0 1522564 1.425g 11712 R 99.6 4.6 10:39.20 bro
32002 root 20 0 1571588 1.472g 11716 R 98.9 4.8 10:36.93 bro
32029 root 20 0 1529472 1.432g 11788 R 98.9 4.7 10:38.50 bro
32016 root 20 0 1341716 1.252g 11712 R 97.9 4.1 10:33.21 bro
32006 root 20 0 1422616 1.330g 11712 R 96.0 4.3 10:20.19 bro

Thanks

2

First verify that you are truly using pf_ring..

$ ldd `which bro`|grep -i pcap
        libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0x00007f70757a6000)

If bro is not actually linked against pf_ring you will end up running 8 workers that are all seeing 100% of the traffic.

While bro is running you should also have a bunch of files in /proc/net/pf_ring/, one for each worker+nic.

3

try deleting the build/ directory in the bro source tree and then running ./configure again with the right options.

As part of the configure output it will say which pcap it is using. We install pf_ring into /opt/pfring so we build bro using

    --with-pcap=/opt/pfring