zbalance_ipc and Zeek

Zeek
1

Hi Bill,
Thank you for the assist. Currently, Zeek cannot reliably capture more than 300Mbps with this configuration. When I remove zbalance_ipc and use RSS with num_rss_queues=lb_procs Zeek can capture up to 2Gbps. I need to use zbalance_ipc because I use a single capture interface with multiple consuming applications, i.e. Zeek and Snort. It seems obvious that a software load balancer will perform less than hardware, however, I don’t see the same significant performance drop with other consuming applications like Snort.

Ingress Line speed:
I am using a traffic generator so I can regulate up to 10Gbps.

ZEEK node.cfg

[manager]
type=manager
host=localhost

[logger]
type=logger
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=zc:99
lb_method=pf_ring
lb_procs=10

pin_cpus=1,2,3,4,5,6,7,8,9,10

ZBALANCE_IPC run config

zbalance_ipc -i zc:eth0 -c 99 -n 10 -m 4 -g 15 -S 0

PFRING-ZC INFO

PF_RING Version : 7.5.0 (unknown)

Total rings : 22

Standard (non ZC) Options

Ring slots : 65536

Slot version : 17

Capture TX : No [RX only]

IP Defragment : No

Socket Mode : Standard

Cluster Fragment Queue : 0

Cluster Fragment Discard : 0

Name : ethØ

Index : 40

Address : XX:XX:XX:XX:XX:XX

Polling Mode : NAPI/ZC

Type : Ethernet

Family : ixgbe

TX Queues : 1

RX Queues : 1

Num RX Slots : 32768

Num TX Slots : 32768

System Specs:
Xeon D-1587 16 cores, 32 logical, 1.7 Ghz, 2.3 Ghz turbo, 20M Cache
128GB DDR4 2133Mhz
8TB SSD
Intel 10GBase-T X557 ixgbe

2

Hi,

I would check the followings :

  • Numa node configuration - This server should have 2 CPU sockets, if you pinned zbalance_ipc to a numa node which is not directly connected to the PCI bus hosting the NIC all traffic will go through the QPI and that could explain why it will be slower. I would check that the zbalance_ipc app is pinned to the CPU socket that is closer to the PCI NIC to avoid this

  • Check line rate on each virtual interface using PF_RING/userland/examples/pfcount. check on : zc:99@[0,1,…,9] after using zbalance_ipc and without zbalance_ipc using the RSS. This should give you a clue if there is a specific worker instance that is receiving significantly more traffic than others (RSS and zbalance_ipc LB might differ). It really depends on the type of traffic, but I assume that on a 2.3Ghz processor, a single bro worker can process anything between 150-400mbps.

  • Run a single instance of bro with local configuration and dump-events.bro script (you can redef include_args=F to get only events name without parameters). Output, sort, uniq -c it to get a clue on what event occur more often. Some analyzers might be turned off to save CPU cycles.
    Let me know if it helps

B

3

Hi Bill,
I just wanted to follow up. I have had success after disabling hyper-threading. I have also isolated the cores for the queue consumers. Zeek and Snort now reliably process over 2Gbps simultaneously with zbalance_ipc. The traffic is a vanilla enterprise profile generated by a traffic generator. I will look into tuning the Zeek analyzers. Thank you for the assist.

CB