Bro 10Gb Performance

Zeek
1

We are currently running Bro with 1 Gb intel cards and vanilla PF_RING and we have acceptable packet loss after filtering (1 – 3 percent), but we need to move up to 10 Gb sensors.

Is there anyone that is using commodity hardware and Intel X520 network cards with Bro to process 10GB of traffic using AF_PACKET, vanilla PF_RING or PF_RING ZC?

In the paper 100G Intrusion Detection, they utilized Myricom 10 Gb cards, with the sniffer software and were only running 10 workers per node or up to 1 Gb per worker. Is this possible on with Intel X520 using AF_PACKET or PF_RING? It is my understanding that AF_PACKET is broken in some kernels (I have used Justin’s fanout tool) and requires a driver update.

Is there a diminishing return for number of workers per server?

Michael Purzynski published a great paper on Suricata performance tuning to achieve 20 Gb throughput on commodity hardware using AF_PACKET. Is there a corresponding Bro document?

Thanks,

Craig Edgmand

IT Security

2

I am using all commodity hardware:
10:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

and I am seeing <5% packet loss. I am currently using Bro 2.5 with AF_PACKET. Bro generally sees on average 7-8Gbps from the taps.

3

I’ve done some research on the use of commodity hardware and the Intel X710 and XL710 cards (Amazon shows ~$475 or so). They are the only non-capture card that allows SR-IOV in promiscuous mode (with on-card vlan popping), which was a requirement for us. AFPacket seems to work well with it (i40e drivers) and passes Justin’s fanout checks on a default RHEL/CentOS 7.3 build).

Did some analysis of the Suricata performance tuning paper WRT bro, and it’s almost universally applicable. Here are some quick recommendations based on some of our analysis:

  • Isolate CPUs in grub (e.g. isolcpus=1-7,9-15)

  • works = CPU Cores - 4. (This leaves 2 cores for the kernel, 1 for proxy, 1 for manager.

  • Lower the Ring Descriptors on the card to ~256 (ethtool -G eth0 rx 256)

  • Reduce “Combined RSS” to 1 (ethtool -L eth0 combined 1)

In the Michael’s paper, he suggests turning off “irqbalance”, but I haven’t seen this make much of a difference, and sometimes hurts more than helps (but I’m also using a single chip system).

Anyway, hope this helps.

~Ed