I have two bro sensors. One is running 2.5, one is running 2.4.1. Both are running on the same link off the tap.
The weird.log on the 2.5 box is 6 times bigger than the weird.log on the 2.4.1 log. Any idea why this might be? How can I troubleshoot this.
My conn.log is 3 times bigger. For reference:
conn.log → 2.5 (45 minutes) 17 gig
conn.log → 2.4.1 (45 min) 5.5 gig
weird.log → 2.5 (45 minutes) 11 gig
weird.log → 2.4.1 (45 minutes) 1.2 gig
These numbers seem to be WAY off. I have no idea how to even try and parse this to see what is going on.
Packet loss on 2.4.1 is 6%
Packet loss on 2.5 is 1%.
Can you take a look at what weirds, specifically, you're getting?
Something like:
cat weird.log | bro-cut name| sort | uniq -c | sort -n
--Vlad
erik clark <philosnef@gmail.com> writes:
Hm. It looks like this may be related to af_packet and bro2.5 in general. I did a subset of production weird and took a subset of development weird, sorted it out and compared the two. From the looks of things, the ratio of items in the files take in identical number of events is pretty close to identical.
This leads me to believe that I am just not dropping traffic either at Bro or the interface on the dev box. Right now I have dropped only 70k packets out of 49TiB of traffic according to ifconfig, and bro is reporting packet loss of ~1%.
The 2.4.1 production box on the other hand is seeing 2-5% packet loss and some packet loss at the interface. The services (http, dns, so on so forth) on the dev box all have equal or more than the number of events on the production box. All I can think of right now is that tuned af_packet on rh7 w/ 2.5 is so much better than tuned pf_ring on rh61 w/ 2.4.1 that it has been noticeable.
Also, memory consumption on 2.5 is a significant fraction less than on the production box with the same link. Wish I could say why this is, but it really impresses me. Load is still high though at ~16, but MEH.
Hmm. I note that I am actually, in a given hour, getting 25-30% less logs from http.log.
Are there any guides to tuning Bro to work with af_packet?
Step 1: ensure that you can use af_packet in the first place:
https://github.com/JustinAzoff/can-i-use-afpacket-fanout/
It looks like your current setup is not working.
Sorry this was supposed to go to the list as well:
Hmm. I see
FAIL: saw flow {tcp $ip $num $ip $num} on workers $num and $num.
This is on RHEL7 with the latest kernel. How can I address what I am assuming is a failure of the kernel?
Yeah, that kernel does not work. I believe Michal said that if you upgrade the ixgb driver to the latest from intel and mess around with ethtool settings you can get it to work.
Justin, any chance you might have that commentary stored anywhere? I spent the past 3 hours trying to find ethtool settings for this, but have had no success. The only thing I did find was
–set-priv-flags $iface rss-symmetric off
but I get no private flags found (for ixgbe nic). Other than that, I can find nothing anywhere concerning this issue that seems to be here.
I did see:
https://media.readthedocs.org/pdf/jasonish-suricata/latest/jasonish-suricata.pdf,
specifically: