Performance Issues after upgrade to Zeek-6

Zeek
7

@redbaron - thank you very much for reporting. I’m able to reproduce something similar in a VM:

  • pf_ring installed via ntop repositories (adds /usr/local/lib to ld.conf setup)
  • lb_procs = 4, lb_method = pf_ring
  • Replaying 10kpps mostly SSL traffic

With Zeek 5.2.2, there’s no packet drops visible through the status files in /proc/net/pf_ring:

root@ubuntu-01:/proc/net/pf_ring# for f in  *enp6s0* ;  do echo "$f $(cat $f | grep Lost) $(cat $f | grep App)" ; done
63289-enp6s0.65 Tot Pkt Lost           : 0 Appl. Name             : zeek-enp6s0
63292-enp6s0.64 Tot Pkt Lost           : 0 Appl. Name             : zeek-enp6s0
63293-enp6s0.63 Tot Pkt Lost           : 0 Appl. Name             : zeek-enp6s0
63294-enp6s0.62 Tot Pkt Lost           : 0 Appl. Name             : zeek-enp6s0

With Zeek 6.0, current observation is that some of the workers immediately or after a short period of tiem start dropping packets:

root@ubuntu-01:/proc/net/pf_ring# for f in  *enp6s0* ;  do echo "$f $(cat $f | grep Lost) $(cat $f | grep App)" ; done
73720-enp6s0.78 Tot Pkt Lost           : 29138 Appl. Name             : zeek-enp6s0
73721-enp6s0.81 Tot Pkt Lost           : 13508 Appl. Name             : zeek-enp6s0
73722-enp6s0.79 Tot Pkt Lost           : 0 Appl. Name             : zeek-enp6s0
73725-enp6s0.80 Tot Pkt Lost           : 0 Appl. Name             : zeek-enp6s0

CPU usage of workers dropping packets plummets.

I’ll look at more detail - this is an obvious regression and I suspect caused by IO loop or timing changes introduced in 6.0.

Thanks again!

Packet loss and compilation issues with PF_RING plugin on Zeek 8.0.6 (3.5 Gbps traffic)