So I am running Bro 2.5.2 in cluster mode using pf_ring and using it to monitor a SPAN port interface. I am running 8 workers and each of them are pinned to a CPU.
When I am performance testing by sending upto 1 gbps of network traffic having a random mix of HTTP, FTP and SMTP data I find that I am getting massive packet loss notices.
{“ts”:1512212763.169748,“note”:“PacketFilter::Dropped_Packets”,“msg”:“4135277 packets dropped after filtering, 4371549 received, 236272 on link”,“peer_descr”:“worker-1-5”,“actions”:[“Notice::ACTION_LOG”],“suppress_for”:3600.0,“dropped”:false}
{“ts”:1512212771.177625,“note”:“PacketFilter::Dropped_Packets”,“msg”:“4827328 packets dropped after filtering, 5073087 received, 245759 on link”,“peer_descr”:“worker-1-7”,“actions”:[“Notice::ACTION_LOG”],“suppress_for”:3600.0,“dropped”:false}
{“ts”:1512212773.214689,“note”:“PacketFilter::Dropped_Packets”,“msg”:“4767851 packets dropped after filtering, 5028737 received, 260886 on link”,“peer_descr”:“worker-1-6”,“actions”:[“Notice::ACTION_LOG”],“suppress_for”:3600.0,“dropped”:false}
{“ts”:1512212783.667576,“note”:“PacketFilter::Dropped_Packets”,“msg”:“5563389 packets dropped after filtering, 5818919 received, 255530 on link”,“peer_descr”:“worker-1-3”,“actions”:[“Notice::ACTION_LOG”],“suppress_for”:3600.0,“dropped”:false}
I am running Bro on a 8 core 8 GB machine with an SSD and not sure why I am getting such high packet loss.
Here is my BroControl netstats and they are also not encouraging.
[BroControl] > netstats
worker-1-1: 1512212665.151426 recvd=297260 dropped=7862632 link=297260
worker-1-2: 1512212659.639980 recvd=251046 dropped=7934351 link=251046
worker-1-3: 1512212652.110004 recvd=261434 dropped=7896026 link=261434
worker-1-4: 1512212662.089539 recvd=291058 dropped=7887963 link=291058
worker-1-5: 1512212666.662180 recvd=246944 dropped=7934732 link=246944
worker-1-6: 1512212661.373981 recvd=254560 dropped=7910802 link=254560
worker-1-7: 1512212657.278461 recvd=255041 dropped=7922435 link=255041
worker-1-8: 1512212671.643251 recvd=214359 dropped=7966526 link=214359
Any help or advise would be greatly appreciated.
Regards,
Vikram Basu