CPU pinning in zeek

Zeek
1

I’m currently using Zeek in clustering mode and discovered that the pin_cpus option allows me to specify which CPU cores the worker nodes should run on. Here is my config

[zeek]
type = logger
host = localhost

[manager]
type = manager
host = localhost

[proxy]
type = proxy
host = localhost

[worker-1]
type = worker
host = localhost
interface = dpdk::0
pin_cpus=5,6,7

I configured worker-1 to use CPU cores 5 through 7, but it always ends up running only on core 5

worker-htop
worker-htop1890×886 49.6 KB

.

Can i ask why it doesn’t run on processor 6 or 7.
Thanks

2

Someone please correct me if I’m wrong since I haven’t done it this way in awhile, but…

I think what you’re seeing is a combination of workers being single threaded and the multiple cpu option w/ ‘pin_cpus’ generally be combined with ‘lb_procs’. I would expect to see a separate worker process per cpu you are pinning.

Something like..
lb_method=custom
lb_procs=8
pin_cpus=0,1,2,3,4,5,6,7

This would cause zeekctl to actually configure and run eight separate workers.

-Dop

1 Like
3

Hi Dopheide,

I tried the configuration you shared — it worked initially, but after about 1–2 minutes, the worker stops automatically. I’m not sure why this happens, but I suspect it might be related to using DPDK as the capture device.

In this setup, is there a specific load balancing method I should be using?

4

Trong,

Does ‘zeekctl diag’ show anything after a worker stops?

-Dop

5 
[worker1-1]                                                                                                                                                                         [33/1918]                                                                                                                                                                                    [32/1918]No core file found.                                                                                                                                                                 [31/1918]                                                                                                                                                                                    [30/1918]Zeek 6.0.2                                                                                                                                                                          [29/1918]Linux 5.14.0-503.40.1.el9_5.x86_64                                                                                                                                                  [28/1918]                                                                                                                                                                                    [27/1918]Zeek plugins:                                                                                                                                                                       [26/1918]ESnet::DPDK - DPDK packet source plugin (dynamic, version 0.1.0)                                                                                                                    [25/1918]ICSNPP::BACnet - BACnet Protocol analyzer (dynamic, version 1.4.0)                                                                                                                  [24/1918]                                                                                                                                                                                    [23/1918]==== No reporter.log                                                                                                                                                                [22/1918]                                                                                                                                                                                    [21/1918]==== stderr.log                                                                                                                                                                     [20/1918]EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL:   using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_vmxnet3 (15ad:7b0) device: 0000:0b:00.0 (socket 0)
EAL: Probe PCI driver: net_vmxnet3 (15ad:7b0) device: 0000:13:00.0 (socket 0)
EAL: No legacy callbacks, legacy socket not created
listening on 0

/opt/zeek/share/zeekctl/scripts/run-zeek: line 110: 725995 Segmentation fault      (core dumped) nohup ${pin_command} $pin_cpu "$myzeek" "$@"

==== stdout.log
max memory size             (kbytes, -m) unlimited
data seg size               (kbytes, -d) unlimited
virtual memory              (kbytes, -v) unlimited
core file size              (blocks, -c) unlimited

==== .cmdline
-i dpdk::0 -U .status -p zeekctl -p zeekctl-live -p local -p worker1-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto -C

==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/root/.local/bin:/root/bin:/opt/thx/sbin:/opt/thx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
ZEEKPATH=/storage/meta/spool/installed-scripts-do-not-touch/site::/storage/meta/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins
CLUSTER_NODE=worker1-1

==== .status
RUNNING [run_loop]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

here it is

when i run dmesg on worker node machine i got this

[228174.061403] zeek[725995]: segfault at 68 ip 00007f4de26aa46e sp 00007fff69f18800 error 4 in ESnet-DPDK.linux-x86_64.so[7f4de26a9000+4000] likely on CPU 0 (core 0, socket 0)
[228174.062810] Code: cd 00 00 00 0f 1f 84 00 00 00 00 00 48 8b bb 38 42 00 00 48 b8 cf f7 53 e3 a5 9b c4 20 48 63 8b 30 42 00 00 4c 01 ef 48 03 0f <48> 8b 71 08 c5 fa 7e 01 c6 45 c0 00 4c 89 75 b0 48 c7 45 b8 00 00
6

Ah, I didn’t realize you were running the ESnet dpdk package. That is currently unmaintained without nobody available to support it. I’d recommend switching to AF_PACKET which is built-in now.

-Dop

1 Like