Packet loss and compilation issues with PF_RING plugin on Zeek 8.0.6 (3.5 Gbps traffic)

[puma2009]

---

Hi everyone,

I’m struggling to get Zeek 8.0.6 to work with PF_RING on a high-traffic sensor. I’m experiencing 80% packet loss at 3.5 Gbps, which is corrupting our file extraction (SMTP/Email).

System Environment:

• OS: cat /etc/os-release → Ubuntu 22.04.3 LTS (Jammy Jellyfish)

• Kernel: uname -a → Linux zeek 5.15.0-101-generic #111-Ubuntu SMP x86_64

• Hardware: 16-core CPU / 64GB RAM.

• Zeek Version: 8.0.6 (Compiled from source in /opt/zeek).

PF_RING Installation Status: I have PF_RING 8.x installed in standard paths, but Zeek is not recognizing it.

• Binaries: /usr/local/bin/pfcount, /usr/local/sbin/pf_ringctl

• Libraries: /usr/local/lib/libpfring.a, /usr/local/lib/libpcap.a (ntop version)

• Headers: /usr/local/include/pfring.h

• **Interfaces listed by PF_RING:
  **
  pf_ringcfg --list-interfaces
  Name: ens18 Driver: virtio_net RSS: 1 [Linux Driver]
  Name: ens16f0 Driver: igb RSS: 8 [Supported by ZC]
  Name: ens16f1 Driver: igb RSS: 8 [Supported by ZC]
  Name: ens16f2 Driver: igb RSS: 8 [Supported by ZC]
  Name: ens16f3 Driver: igb RSS: 8 [Supported by ZC]

  cat /proc/net/pf_ring/dev/ens16f0/info
  Name: ens16f0
  Index: 3
  Address: xxxxxxxxxx
  Polling Mode: NAPI
  Promisc: Disabled
  Type: Ethernet
  Family: Standard NIC
  bound sockets: 0

  TX Queues: 8
  RX Queues: 8
  my node:

  logger

  type=logger
  host=localhost

  manager

  type=manager
  host=localhost

  proxy-1

  type=proxy
  host=localhost

  worker-1

  type=worker
  host=localhost
  interface=ens16f0
  lb_method=pf_ring
  lb_procs=4
  pin_cpus=0,1,2,3

  lb_param=10

  worker-2

  type=worker
  host=localhost
  interface=ens16f1
  lb_method=pf_ring
  lb_procs=4
  pin_cpus=4,5,6,7
  lb_param=20

  worker-3

  type=worker
  host=localhost
  interface=ens16f2
  lb_method=pf_ring
  lb_procs=3
  pin_cpus=8,9,10
  lb_param=30

  worker-4

  type=worker
  host=localhost
  interface=ens16f3
  lb_method=pf_ring
  lb_procs=3
  pin_cpus=11,12,13
  lb_param=40

  capstats:

  Interface kpps mbps (10s average)

  localhost/ens16f0 182.5 909.7
  localhost/ens16f1 173.5 918.0
  localhost/ens16f2 129.8 718.5
  localhost/ens16f3 170.7 779.3

  Total 656.5 3325.5

  The Wall:

  1. Plugin 404: The repository https://github.com/ntop/zeek-plugin-pf_ring is returning a 404, and it’s no longer present in the PF_RING/userland source tree.

  2. zkg failure: Running zkg install zeek-plugin-pf_ring returns “package name not found in sources”.

  3. Core Build: Compiling Zeek with --with-pcap=/usr/local finishes successfully, but zeek -N does not list Zeek::PF_RING.

  Questions for the community:

  1. The “Best Practice” 2026 Question: For 3.5 Gbps on a modern Linux Kernel (5.15+), is PF_RING (non-ZC) still superior to AF_PACKET + Fanout? Or has AF_PACKET become the recommended non-paid path for these speeds?

  2. The Plugin: If PF_RING is still the way to go, where is the official source code for the Zeek 8 plugin located now?

  3. Kernel Bottleneck: Is there any known issue with the 5.15 kernel and Zeek’s file extraction that could be exacerbated by the current packet loss?

  Any help to break this loop and get the sensor back to zero-drop would be greatly appreciated.

[awelzel]

Hey @puma2009,

not a PF_RING expert, but I’d expect it to work. The idea is that Zeek is linked against a PF_RING specific libpcap version as you showed here:

Core Build: Compiling Zeek with --with-pcap=/usr/local finishes successfully, but zeek -N does not list Zeek::PF_RING.

I’m not aware of the need for a plugin. So Zeek::PF_RING isn’t expected to show up. There’s an older bro-pf_ring plugin, but that should not be necessary. Please check with ldd /opt/zeek/bin/zeek to see if Zeek is dynamically linked against PF_RING’s specific libpcap.so file. If not, then the cluster might currently use default libpcap and load-balancing between worker not in effect, easily resulting in overload.

Further, do you have PFRINGClusterType in your zeekctl.cfg? It defaults to 4-tuple, but inner-5-tuple or inner-6-tuple may be more reasonable today when dealing with tunneled traffic.

If you look at zeekctl top, is there one or two workers at 100% usage, or are all of them at 100% usage. In the former case, flows might end-up only on a few processes. In the latter case, processes might all see all flows rather than just a portion.

I’m a bit confused about the lb_param value in your node.cfg - I don’t think this exists. Is it something custom?

The “Best Practice” 2026 Question: For 3.5 Gbps on a modern Linux Kernel (5.15+), is PF_RING (non-ZC) still superior to AF_PACKET + Fanout? Or has AF_PACKET become the recommended non-paid path for these speeds?

The existing PF_RING setup should continue to work. Please double check the libpcap.so file in use by Zeek.

I do think AF_PACKET is easier to setup, so it might be easy for you to do a quick comparison.

With Zeek 8.0, AF_PACKET is builtin on Linux, so you could try:

[worker-1]
type=worker
host=localhost
interface=ens16f2
lb_method=af_packet
af_packet_fanout_id = 12
...

Might make sense to search for offloading and tuning related to AF_PACKET. Also, with multiple network cards on a single system, make sure you set af_packet_fanout_id to a unique value and double check on the block/buffer size tunables.

1. Kernel Bottleneck: Is there any known issue with the 5.15 kernel and Zeek’s file extraction that could be exacerbated by the current packet loss?

This should be fine

Any help to break this loop and get the sensor back to zero-drop would be greatly appreciated.

Hope this helps. Curious if you can figure out more. Please share if you determine the culprit!

Thanks,
Arne

[awelzel]

@puma2009 - which PF_RING version are you using? There was an issue around poll/epoll reported ~2.5years ago when upgrading to Zeek 6.0 that we reported to PF_RING. It was with PF_RING 8.2.0 and the fix made it into PF_RING 8.8.0 or later.

Alternatively, a fix in Zeek would be to try and put the following into local.zeek to poll PF_RING more often:

# local.zeek - Use more aggressive polling to avoid polling issue with kqueue and PF_RING < 8.8.0.
redef io_poll_interval_live = 100;

So if you don’t yet have PF_RING 8.8.0 or later, try upgrading or see if the above redef helps.

Please report back if any of these steps helps.

[puma2009]

Hi @awelzel, thanks for the advice. It turns out I’ve noticed that the spans on one of our Gigamon power supplies are at 100%, and this might be one of the reasons why the TCP sessions aren’t reassembling correctly in Zeek. I’m going to fix the problems with this power supply first, and then I’ll make the necessary changes in Zeek. I’ll let you know which solution is better in my case: AF_PACKET or a fine-tuned libpcap configuration with pf_ring. Thanks a lot, we’ll stay in touch until we resolve the span issues.

[awelzel]

Hello @puma2009- just checking in if you resolved the packet loss issues and possibly have any interesting findings to share?

Thanks, Arne

[puma2009]

Hello again, thanks for all,

I have partially resolved the massive flow from gigamon through the 4 spans and now it is at better, albeit high, levels.

tail -n20 /data/zeek/logs/current/capture_loss.log
1775571475.862498 900.000347 worker-1-3 767710 1704332 45.044627
1775571475.940897 900.000041 worker-1-1 760241 1687403 45.053908
1775571475.480382 900.001275 worker-3-1 151746 736781 20.595808
1775571475.386703 900.001676 worker-4-2 526689 723603 72.787012
1775571475.503528 900.000042 worker-3-2 179596 784620 22.889552
1775571475.806397 900.000038 worker-4-1 451037 649138 69.482452
1775572375.469437 900.000026 worker-2-2 10749 3465621 0.310161
1775572375.536061 900.000391 worker-2-1 11189 3017828 0.370763
1775572375.941246 900.000349 worker-1-1 797014 1758403 45.326015
1775572375.821387 900.000016 worker-1-5 771190 1780335 43.317129
1775572375.822714 900.000038 worker-1-2 837227 1911406 43.801631
1775572375.828865 900.000157 worker-1-4 700786 1643333 42.644187
1775572375.767111 900.000546 worker-1-7 689981 1612009 42.802553
1775572375.921164 900.000123 worker-1-8 694628 1567299 44.320069
1775572375.856049 900.000060 worker-1-6 676149 1519964 44.48454
1775572375.862603 900.000105 worker-1-3 743189 1737754 42.767216
1775572375.812418 900.006021 worker-4-1 442786 639100 69.282741
1775572375.480386 900.000004 worker-3-1 129748 777420 16.689563
1775572375.503630 900.000102 worker-3-2 153172 941941 16.261316
1775572375.391850 900.005147 worker-4-2 514047 697742 73.672934

I used to lose around 90%, now it’s around 40%. I changed the configuration to AF packet:

cat /usr/local/zeek/etc/node.cfg

logger

type=logger
host=localhost
pin_cpus=0

manager

type=manager
host=localhost
pin_cpus=1

proxy-1

type=proxy
host=localhost
pin_cpus=1

— INTERF 1 (8 workers) —

worker-1

type=worker
host=localhost
interface=af_packet::ens16f0
lb_method=custom
lb_procs=8
pin_cpus=2,3,4,5,6,7,8,9
af_packet_fanout_id=77
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=1073741824

— INTERF 2 (2 workers) —

worker-2

type=worker
host=localhost
interface=af_packet::ens16f1
lb_method=custom
lb_procs=2
pin_cpus=10,11
af_packet_fanout_id=20
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=134217728

— INTERF 3 (2 workers) —

worker-3

type=worker
host=localhost
interface=af_packet::ens16f2
lb_method=custom
lb_procs=2
pin_cpus=12,13
af_packet_fanout_id=30
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=134217728

— INTERF 4 (2 workers) —

worker-4

type=worker
host=localhost
interface=af_packet::ens16f3
lb_method=custom
lb_procs=2
pin_cpus=14,15
af_packet_fanout_id=40
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=134217728

my zeek conf:

zeek -V
{
“branch”: “release/7.0”,
“commit”: “f300ddb9fe018fe26209310e13ea2f7e4eb95702”,
“describe”: “v7.0.4”,
“dirty”: false,
“included_plugins”: [
{
“name”: “zeek-af_packet-plugin”,
“version”: “4.0.0”
}
],
“name”: “zeek”,
“source”: “repo-info.json”,
“submodules”: [
{
“commit”: “7c5ccc9aa91466004bc4a0dbbce11a239f3e742e”,
“describe”: “v1.8.0-19-g7c5ccc9”,
“dirty”: false,
“path”: “auxil/bifcl”,
“version”: “1.8.0-16”
},
{
“commit”: “a5c8f19fb49c60171622536fa6d369fa168f19e0”,
“describe”: “v0.61.0-31-ga5c8f19”,
“dirty”: false,
“path”: “auxil/binpac”,
“version”: “0.61.0-28”
},
{
“commit”: “a80bf420aa6f55b4eb959ae89c184522a096a119”,
“describe”: “v2.8.0-dev-96-ga80bf420”,
“dirty”: false,
“path”: “auxil/broker”,
“version”: “2.8.0-dev.97”
},
{
“commit”: “989c7513c3b6056a429a5d48dacdc9a2c1b216a7”,
“describe”: “v1.1-8-g989c751”,
“dirty”: false,
“path”: “auxil/btest”,
“version”: “1.1-8”
},
{
“commit”: “0ad09d251bf01cc2b7860950527e33e22cd64256”,
“describe”: “cares-1_26_0”,
“dirty”: false,
“path”: “auxil/c-ares”
},
{
“commit”: “72a76d774e4c7c605141fd6d11c33cc211209ed9”,
“describe”: “v1.1.2-405-g72a76d7”,
“dirty”: false,
“path”: “auxil/filesystem”
},
{
“commit”: “610cf8527dad7033b971595a1d556c2c95294f2b”,
“describe”: “heads/master”,
“dirty”: false,
“path”: “auxil/gen-zam”,
“version”: “1.0.0”
},
{
“commit”: “5ad3bf8444cfc663b11bf367baaa31f36e7ff7c8”,
“describe”: “5ad3bf8”,
“dirty”: false,
“path”: “auxil/highwayhash”
},
{
“commit”: “10d93cff9fd6c8d8c3e0bae58312aed470843ff8”,
“describe”: “v2.4.0-291-g10d93cf”,
“dirty”: false,
“path”: “auxil/libkqueue”
},
{
“commit”: “b38e9c8ebff08959a712a5663ba25e0624a3af00”,
“describe”: “heads/master”,
“dirty”: false,
“path”: “auxil/libunistd”
},
{
“commit”: “e7643a319d36fc32573cb99b363040342f9d8e03”,
“describe”: “v0.6-2-ge7643a3”,
“dirty”: false,
“path”: “auxil/netcontrol-connectors”,
“version”: “0.6”
},
{
“commit”: “ea379b2f35e28d6ee894e05ad4c26ed60a613d30”,
“describe”: “ea379b2”,
“dirty”: false,
“path”: “auxil/out_ptr”
},
{
“commit”: “bdc15fab95b1ca2bd370fa25d91f7879b5da35fc”,
“describe”: “v3.0.1-33-gbdc15fa”,
“dirty”: false,
“path”: “auxil/package-manager”,
“version”: “3.0.1-33”
},
{
“commit”: “45ce017874aac9ffabac0ddc4d016f1747804234”,
“describe”: “v0.6.0-11-g45ce017”,
“dirty”: false,
“path”: “auxil/paraglob”,
“version”: “0.6.0-8”
},
{
“commit”: “ad99e21f4706193670c42b36c9824dc997f4c475”,
“describe”: “ad99e21”,
“dirty”: false,
“path”: “auxil/prometheus-cpp”
},
{
“commit”: “6089180ecb704cb2b136777798fa1be303618975”,
“describe”: “v1.1.0-745-g6089180e”,
“dirty”: false,
“path”: “auxil/rapidjson”
},
{
“commit”: “31880e215ff6f85ad95507d7a760e81ae2e0999d”,
“describe”: “v1.11.3”,
“dirty”: false,
“path”: “auxil/spicy”,
“version”: “1.11.3”
},
{
“commit”: “66b4b34d99ab272fcf21f2bd12b616e371c6bb31”,
“describe”: “2022.02.02-5950-g66b4b34d9”,
“dirty”: false,
“path”: “auxil/vcpkg”
},
{
“commit”: “a3fe59b3f1ded5c3461995134b66c6db182fa56f”,
“describe”: “4.0.0-24-ga3fe59b”,
“dirty”: false,
“path”: “auxil/zeek-af_packet-plugin”,
“version”: “4.0.0”
},
{
“commit”: “e850412ab5dea10ee2ebb98e42527d80fcf9a7ed”,
“describe”: “v0.50-140-ge850412”,
“dirty”: false,
“path”: “auxil/zeek-aux”,
“version”: “0.50-137”
},
{
“commit”: “5bcc14085178ed4ddfa9ad972b441c36e8bc0787”,
“describe”: “v1.3.1-21-g5bcc140”,
“dirty”: false,
“path”: “auxil/zeek-client”,
“version”: “1.3.1-21”
},
{
“commit”: “39c0ee1e1742bb28dff57632ee4620f905b892e7”,
“describe”: “v2.5.0-53-g39c0ee1”,
“dirty”: false,
“path”: “auxil/zeekctl”,
“version”: “2.5.0-49”
},
{
“commit”: “d5bae4c4d29e3c5e87d54f94d7a9b3f86ae856a8”,
“describe”: “v0.13.2”,
“dirty”: false,
“path”: “auxil/zeekjs”,
“version”: “0.13.2”
},
{
“commit”: “2d42baf8e63a7494224aa9d02afa2cb43ddb96b8”,
“describe”: “release-338-g2d42baf”,
“dirty”: false,
“path”: “cmake”
},
{
“commit”: “cafe6ac085091ad36bca74c8fc18ff6c1f4cf9fd”,
“describe”: “v7.0.4”,
“dirty”: false,
“path”: “doc”
},
{
“commit”: “96caa6a59023eafae8a94a7b906a4436ebb57d29”,
“describe”: “release-100-g96caa6a”,
“dirty”: false,
“path”: “src/3rdparty”
}
],
“version”: “7.0.4”,
“zkg”: {
“provides”: [
{
“name”: “zeek-af_packet-plugin”,
“version”: “4.0.0”
},
{
“name”: “spicy-plugin”,
“version”: “7.0.4”
}
]

and the status:

zeekctl status
Warning: zeekctl config has changed (run the zeekctl “deploy” command)
Name Type Host Status Pid Started
logger logger localhost running 4072286 07 Apr 00:01:47
manager manager localhost running 4072352 07 Apr 00:01:50
proxy-1 proxy localhost running 4072405 07 Apr 00:01:52
worker-1-1 worker localhost running 4072642 07 Apr 00:01:53
worker-1-2 worker localhost running 4072645 07 Apr 00:01:53
worker-1-3 worker localhost running 4072658 07 Apr 00:01:53
worker-1-4 worker localhost running 4072639 07 Apr 00:01:53
worker-1-5 worker localhost running 4072643 07 Apr 00:01:53
worker-1-6 worker localhost running 4072627 07 Apr 00:01:53
worker-1-7 worker localhost running 4072638 07 Apr 00:01:53
worker-1-8 worker localhost running 4072664 07 Apr 00:01:53
worker-2-1 worker localhost running 4072661 07 Apr 00:01:53
worker-2-2 worker localhost running 4072663 07 Apr 00:01:53
worker-3-1 worker localhost running 4072651 07 Apr 00:01:53
worker-3-2 worker localhost running 4072662 07 Apr 00:01:53
worker-4-1 worker localhost running 4072657 07 Apr 00:01:53
worker-4-2 worker localhost running 4072665 07 Apr 00:01:53

eekctl top
Warning: zeekctl config has changed (run the zeekctl “deploy” command)
Name Type Host Pid VSize Rss Cpu Cmd
logger logger localhost 4072286 5G 287M 55% zeek
manager manager localhost 4072352 1G 252M 5% zeek
proxy-1 proxy localhost 4072405 1G 264M 27% zeek
worker-1-1 worker localhost <unexpected top output: could not convert string to float: ‘4,1g’>
worker-1-2 worker localhost <unexpected top output: could not convert string to float: ‘4,1g’>
worker-1-3 worker localhost <unexpected top output: could not convert string to float: ‘4,2g’>
worker-1-4 worker localhost <unexpected top output: could not convert string to float: ‘4,2g’>
worker-1-5 worker localhost <unexpected top output: could not convert string to float: ‘4,4g’>
worker-1-6 worker localhost <unexpected top output: could not convert string to float: ‘4,2g’>
worker-1-7 worker localhost <unexpected top output: could not convert string to float: ‘4,2g’>
worker-1-8 worker localhost <unexpected top output: could not convert string to float: ‘4,2g’>
worker-2-1 worker localhost 4072661 1G 628M 38% zeek
worker-2-2 worker localhost 4072663 1G 645M 72% zeek
worker-3-1 worker localhost <unexpected top output: could not convert string to float: ‘5,4g’>
worker-3-2 worker localhost <unexpected top output: could not convert string to float: ‘5,3g’>
worker-4-1 worker localhost <unexpected top output: could not convert string to float: ‘6,7g’>
worker-4-2 worker localhost <unexpected top output: could not convert string to float: ‘7,2g’>

hardware specifications:

lscpu | grep ‘^CPU(s):’ && free -h | grep ‘^Mem:’
CPU(s): 16
Mem: 118Gi 66Gi 7.5Gi 1.0Mi 45Gi 51Gi

drop packets:

for i in /sys/class/net/ens16f*; do iface=$(basename $i); echo -n "$iface: "; ip -s link show $iface | awk ‘/RX:/ {getline; rx=$1; getline; drop=$3; printf “RX: %.2f TB | Drops: %s\n”, rx/1024/1024/1024/1024, drop}’; done
ens16f0: RX: 66.10 TB | Drops: packets
ens16f1: RX: 270.27 TB | Drops: packets
ens16f2: RX: 379.25 TB | Drops: packets
ens16f3: RX: 347.24 TB | Drops: packets

for i in /sys/class/net/ens16f*; do iface=$(basename i); model=(lspci | grep -i "$(ethtool -i $iface | awk '/bus-info/ {print 2}' | cut -d: -f2-)" | cut -d: -f3- | sed 's/^ //'); speed=(ethtool $iface | awk '/Speed:/ {print 2}'); stats=(ip -s link show $iface | awk ‘/RX:/ {getline; rx=$1; getline; drop=$3; printf “RX: %.2f TB | Drops: %s”, rx/1024/1024/1024/1024, drop}’); echo -e “Interface: $iface\n Model: $model\n Speed: $speed\n Traffic: $stats\n”; done
Interface: ens16f0
Model: Intel Corporation I350 Gigabit Network Connection (rev 01)
Speed: 1000Mb/s
Traffic: RX: 66.10 TB | Drops: packets

Interface: ens16f1
Model: Intel Corporation I350 Gigabit Network Connection (rev 01)
Speed: 1000Mb/s
Traffic: RX: 270.27 TB | Drops: packets

Interface: ens16f2
Model: Intel Corporation I350 Gigabit Network Connection (rev 01)
Speed: 1000Mb/s
Traffic: RX: 379.25 TB | Drops: packets

Interface: ens16f3
Model: Intel Corporation I350 Gigabit Network Connection (rev 01)
Speed: 1000Mb/s
Traffic: RX: 347.25 TB | Drops: packets

I’m thinking of switching to the free version of pfring to see if it improves the loss rate.