I have Bro 2.4 configured on a RHEL 6.8 server and was wondering how to properly configure the network interfaces so that Bro can see as much of the network traffic as possible. My tap is connected in line with the network, and I believe that I was previously seeing the correct traffic, but now Bro has reporting much less information. I want to make sure that I have the interfaces configured correctly before moving on to troubleshooting other areas. Currently, I have two eth interfaces set up in PROMISC mode. Thank you for the help
There is one further thing I would recommend though that we discovered well after this blog post was written. If you are using an Intel NIC with the ixgbe driver, your nic has a feature called "flow director" that you will want to disable because it will negatively impact your analysis by reordering packets. It can be disabled like this on linux:
ethtool -L eth12 combined 1
This will cause your NIC to have only a single hardware queue which will disable the flow director feature and prevent your NIC from reordering packets. Do that along with the suggestions in the blog post above and things should be better.
Thank you for the help. I tried the settings, but I have noticed any difference in packets. The main test that I am doing is that I would open two putty sessions to the server, and have one running capstats on eth12 while my other session was downloading a 1GB file to /dev/null. Last week, I was able to see the packets increase greatly via capstats, but now they stay steady at 7 or 8 packets per second.
This is what I use in my sensor /etc/network/interfaces config along with a custom “post-up” script. I use Debian for my Bro clusters, so your application will differ. I’m also using af_packet (v4.8.0 kernel) so some of the performance settings may need to be adjusted for your setup. My tuning is aimed at keeping the packets in L3 cache on the CPU vid the NIC hardware, hence the reduced rings.
auto eth6
iface eth6 inet manual
up ip link set $IFACE promisc on arp off mtu 1500 up
down ip link set $IFACE promisc off down
post-up /opt/tools/post-up_settings.sh $IFACE
And the /opt/tools/post-up_settings.sh script:
#!/bin/bash
IFACE=$1
if [[ -n “$IFACE” ]]; then
Lower the NIC ring descriptor size
/sbin/ethtool -G $IFACE rx 512
Disable offloading functions
for i in rx tx sg tso ufo gso gro lro rxhash ntuple txvlan rxvlan; do ethtool -K $IFACE $i off; done
Enforce a single RX queue
/sbin/ethtool -L $IFACE combined 1
Disable pause frames
/sbin/ethtool -A $IFACE rx off tx off
Limit the maximum number of interrupts per second
/sbin/ethtool -C $IFACE adaptive-rx on rx-usecs 100
So I’m not sure I follow exactly why you’d want to specifically emphasize keeping packets in the L3 cache. Is there a specific hardware configuration where this makes more sense?
As of right now, I do pretty much the same thing you posted earlier except I map the # of RX queues to the # of physical CPU cores and maximize the NIC ring descriptor size.
We’ve been tracking the “Suricata Extreme Performance Tuning” efforts where they’re hitting 20Gbps on a single box. They have a pretty good write-up of their research into the flow of a packet through the hardware/software layers: https://github.com/pevma/SEPTun
The RSS=1 setting is to avoid packet re-ording impacts. Are you using PF_RING DNA with symmetric RSS?
One new issue we’re working on since switching from pf_ring to af_packet is that the CPU that worker-1 is pinned to runs close to 100% while the 9 other workers are all much lower than that. We’re been able to determine that its due to the Intel ‘set_irq_affinity’ script and its pinning every IRQ call for the NIC (and single RSS to the same CPU, which is the first core in that NUMA node.
We’re always looking for tuning feedback/best practices so I appreciate your questions.