Hello,
I'm developing a program that dumps selected traffic to a tap device
which bro is listening on. This works great. However to increase
performance I'm trying to use receive side scaling, splitting the packet
stream into multiple queues according to ips, ports etc. This results in
packet reordering which confuses bro and the data is not analyzed and
assembled correctly. Other programs such as wireshark and tcpflow are
able to assemble the traffic correctly so all data is there. Typically
small packets such as acks seems to arrive before larger packets.
I have been searching for bro configurations that affect the tcp
reassembly process but have so far not found anything that makes the
situation better. Is there any particular configurations I should look at?
Anyone have experience with RSS and have any ideas how the packet
reordering issue can be mitigated?
Thank you
/Björn
(DoNotAddDisclaimer)
Are you using straight RSS without any modifications? By default the Toeplitz hash that RSS uses is not a bidirectional hash. i.e., each flow of a connection ends up on a different NIC queue. As a general rule though, you should try to avoid anything that could result in packet reordering.
Another question, how are you making Bro attach to the various NIC hardware queues?
.Seth
Hello Seth,
Thanks for the answer.
I'm using pfring with rss rehashing so it's bidirectional.
From bor's perspective there is only a tap device with one queue . I'm
developing a program that basically acts as a filter between the network
stream and bro, this program uses RSS to be able to cope with 10GBit/s.
Bro only sees a small fraction of the entire network stream.
I would like to disable the RSS which is working extremely well.
Unfortunately this gives me performance issues since I'm not able to
process all the traffic.
So potential solutions I'm investigating.
1. In some way mitigate the reordering issue at network card/pfring
level without disabling RSS, have so far not found a solution for this.
2. Tell bro to reorder the packets, I guess bro already does this but
gives up if the packets are to much out of order. Is there any
parameters I can change to tell bro to try harder?
3. One option is of cause to reorder the packets myself before sending
them to bro.
Thanks
Björn
(DoNotAddDisclaimer)
I'm using pfring with rss rehashing so it's bidirectional.
Ahh, that explains a lot.
From bor's perspective there is only a tap device with one queue . I'm
developing a program that basically acts as a filter between the network
stream and bro, this program uses RSS to be able to cope with 10GBit/s.
Bro only sees a small fraction of the entire network stream.
You may want to wait a bit on this. 
We should be announcing a new tool soon that does this and is *very* flexible. It uses netmap to get packets into user space and then you essentially get "packet bricks" to build your own packet handling pipeline, very similar to the "Click! Software Router" if you're familiar with that. Our intern this summer has been working really hard on getting this tool ready and I think that everyone is really going to like it.
1. In some way mitigate the reordering issue at network card/pfring
level without disabling RSS, have so far not found a solution for this.
With the way that PF_Ring is doing this I don't think there is a solution for it.
2. Tell bro to reorder the packets, I guess bro already does this but
gives up if the packets are to much out of order. Is there any
parameters I can change to tell bro to try harder?
This is kind of mess and has a ton of edge cases since the packets arrive without timestamps we wouldn't even have a mechanism to reorder them. The TCP reassembler ends up being how they're reordered but that can fail for a number of reasons due to the reordering.
3. One option is of cause to reorder the packets myself before sending
them to bro.
Yes, but you're in the position where you still wouldn't know how to reorder them without timestamps.
.Seth
Interesting! Will this tool be flexible enough to also feed packets to other programs on the same box such as argus, snort, etc?
Cheers,
Jesse
Yep, that was a specific design requirement. It uses the newish netmap feature "pipes" to give you specially named interfaces that you can open in anything using the netmap libpcap support.
.Seth