# Update on using PF\_RING/TNAPI with Bro **URL:** \ **Category:** Zeek\ **Created:** [December 15, 2010, 9:41pm UTC](https://community.zeek.org/t/update-on-using-pf-ring-tnapi-with-bro/1769 "2010-12-15T21:41:36Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Sunjeet\_Singh](https://avatars.discourse-cdn.com/v4/letter/s/ecb155/32.png) [@Sunjeet\_Singh](https://community.zeek.org/u/Sunjeet_Singh)\ **Post date:** [December 15, 2010, 9:41pm UTC](https://community.zeek.org/t/update-on-using-pf-ring-tnapi-with-bro/1769/1 "2010-12-15T21:41:36Z")
Hi, I have managed to get TNAPI/PF\_RING configured and working with PF\_RING-aware libpcap. [http://www.ntop.org/TNAPI.html](http://www.ntop.org/TNAPI.html) Looks like this will be very well suited to the Multiprocessing version of Bro. 1. At the device driver level, RSS functionality (also, Flow Director in Intel) allows packets multiplexed to different Receiver Queues (and also allows packets belonging to a particular connection be sent to the same RX\_Queue) on an I/OAT-supporting network card. 2. By virtue of TNAPI, these multiple RX\_Queues get polled concurrently (by one kernel thread per queue), and sent to PF\_RING (along with information about which queue the packet came from). 3. PF\_RING provides a user API which can be used by user-applications like Bro to directly read from the multiple RX\_Queues of a network interface by using notation like eth0@1, eth0@2, etc. for RX\_Queues 1 and 2 belonging to interface eth0. By assigning one thread to one RX\_Queue, we ensure that all packets from one connection are being processed by the same core. PF\_RING and TNAPI can be used to drastically improve the performance of any multiprocessing application, but need to be properly tuned and used by the application. Performance stems from the fact that for Bro, the packets can bypass the kernel’s network stack altogether; one thread polling per RX\_Queue thanks to TNAPI; and PF\_RING avoiding the mmap from Kernel space to User space by directly copying payloads from RX\_Queue rings. Configuration wise, it took a bit of work to change Bro’s configure files to use a PF\_RING-aware libpcap instead of the libpcap that Bro ships with. When running TNAPI and PF\_RING, there is a clear performance improvement in the kernel’s ability to receive packets at a higher packet rate (results on the TNAPI website, I also verified). But using PF\_RING with the existing Bro leads to a performance degradation of Bro because Bro runs on one user-thread, and when all these packets reach user-space on different user-threads, they need to be processed by the core that is running Bro. But from my knowledge on TNAPI/PF\_RING and intuition, multi-threaded Bro can be adapted to PF\_RING and will lead to huge gains in performance. Here’s the summary of results of a brief experiment that I performed on a 8-core Intel Xeon with32 GB RAM running on Linux and with an Intel 82598EB 10Gbps ethernet card: Goal: Compare conventional Bro installation against Bro with TNAPI and PF\_RING (I called it Bro-Ring) Conclusion: Bro-Ring shows a performance drop. Observations: The values in the table show for varying packet-rates, how many packets were accepted by the machine running Bro (rest were lost). > Packets/sec | Bro-Ring | Bro | > > - | - | - | > 34000 | 1368791 | 1368003 | > 50000 | 1368546 | 1367707 | > 65,000 | 1368614 | > | > 120000 | > | 1224761 | > 130000 | > | 1168734 | > 166000 | 596667 | > | > 170000 | 561702 | > | > 171000 | 681104 | > | > 173000 | 618100 | > | > 175000 | 740137 | > | > 178000 | 864706 | > | > 210000 | > | 753700 | > 215000 | > | 728450 | > 230000 | 494637 | > | > 240000 | > | 636287 | (Note: there was a difference in tcpreplay’s input parameter packet-rate and the actual packet rate achieved, so I could not supply exact values for packet rate) Sunjeet Singh ---
**Author:** ![Sunjeet\_Singh](https://avatars.discourse-cdn.com/v4/letter/s/ecb155/32.png) [@Sunjeet\_Singh](https://community.zeek.org/u/Sunjeet_Singh)\ **Post date:** [December 15, 2010, 9:50pm UTC](https://community.zeek.org/t/update-on-using-pf-ring-tnapi-with-bro/1769/2 "2010-12-15T21:50:41Z")
The table got a bit messed up, here are the values again- Packets/sec Bro-Ring Bro 34000 13687911 36800 350000 13685461 367707 65000 1368614 - 120000 - 1224761 130000 - 1168734 166000 596667 - 170000 561702 - 171000 681104 - 173000 618100 - 175000 740137 - 178000 864706 - 210000 - 753700 215000 - 728450 230000 494637 - 240000 - 636287 Sunjeet ---
**Author:** ![Justin\_Azoff](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@Justin\_Azoff](https://community.zeek.org/u/Justin_Azoff)\ **Post date:** [December 15, 2010, 10:00pm UTC](https://community.zeek.org/t/update-on-using-pf-ring-tnapi-with-bro/1769/3 "2010-12-15T22:00:17Z")
> 3. PF\_RING provides a user API which can be used by user-applications > like Bro to directly read from the multiple RX\_Queues of a network > interface by using notation like eth0@1, eth0@2, etc. for RX\_Queues 1 > and 2 belonging to interface eth0. ... > But using PF\_RING with the existing Bro leads to a performance > degradation of Bro because Bro runs on one user-thread, and when all > these packets reach user-space on different user-threads, they need to > be processed by the core that is running Bro. Why are you only running one bro process? You can setup a single node bro cluster and run multiple bro processes, each listening on one of eth0@1, eth0@2.. ---
**Author:** ![Sunjeet\_Singh](https://avatars.discourse-cdn.com/v4/letter/s/ecb155/32.png) [@Sunjeet\_Singh](https://community.zeek.org/u/Sunjeet_Singh)\ **Post date:** [December 15, 2010, 10:11pm UTC](https://community.zeek.org/t/update-on-using-pf-ring-tnapi-with-bro/1769/4 "2010-12-15T22:11:18Z")
> Why are you only running one bro process? You can setup a single node > bro cluster and run multiple bro processes, each listening on one of > eth0@1, eth0@2.. Yes, that's a great idea. But I'm not sure how Bro would handle manager-proxy-worker communication between different RX\_Queues instead of different interfaces. Can't be as simple as writing eth0@1, etc. in the cluster's node.cfg file. Maybe some changes to Bro code? Sunjeet ---
**Author:** ![Justin\_Azoff](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@Justin\_Azoff](https://community.zeek.org/u/Justin_Azoff)\ **Post date:** [December 16, 2010, 3:14am UTC](https://community.zeek.org/t/update-on-using-pf-ring-tnapi-with-bro/1769/5 "2010-12-16T03:14:47Z")
Putting eth0@1,2,3,4 ec. in node.cfg should work just fine. no changes to bro are needed, but you may have to rebuild bro with ./configure --enable-cluster... the config I use with click just has: [manager] type=manager host=10.10.1.12 [proxy-1] type=proxy host=10.10.1.12 [worker-1] type=worker host=10.10.1.12 interface=tap0 [worker-2] type=worker host=10.10.1.12 interface=tap1 [worker-3] type=worker host=10.10.1.12 interface=tap2 [worker-4] type=worker host=10.10.1.12 interface=tap3 ---
**Author:** ![Sunjeet\_Singh](https://avatars.discourse-cdn.com/v4/letter/s/ecb155/32.png) [@Sunjeet\_Singh](https://community.zeek.org/u/Sunjeet_Singh)\ **Post date:** [December 17, 2010, 1:30am UTC](https://community.zeek.org/t/update-on-using-pf-ring-tnapi-with-bro/1769/6 "2010-12-17T01:30:40Z")
Okay, if this works, I don't think you'll see a gain in performance. To leverage performance in this case, lets say you have 4 cores, Core 0 running Bro Manager, Core 1 running Bro Proxy, Core 2 running Bro Worker1, Core 3 running Bro Worker2. For max performance (and really any performance gain) through cache localization, you'd want all traffic input to Bro to go to either Core2 or Core3; and both of these cores coupled to one RX\_Queue each. You will somehow need to at the driver layer, split the traffic coming from the wire to go to one of these queues (intelligently to send packets sharing state to the same RX\_Queue). This has to be done at the RSS, and I have no idea of how to do this on my network card- Intel 82598EB. (You couldn't use Click to do this because you want to do it at the driver level). What do you think? Sunjeet ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:39pm UTC](https://community.zeek.org/t/update-on-using-pf-ring-tnapi-with-bro/1769/7 "2022-05-06T15:39:22Z")
Closed on May 6, 2022, 3:39pm UTC