Hey all,
I’m having a really tough time getting PF_RING working with Bro in a threaded fashion. I have PF_RING compiled and working fine (tcpdump test works fine with Transparent mode = 2):
PF_RING Version : 6.0.2 ($Revision: exported$)
Total rings : 0
Standard (non DNA) Options
Ring slots : 4096
Slot version : 16
Capture TX : No [RX only]
IP Defragment : No
Socket Mode : Standard
Transparent mode : No [mode 2]
Total plugins : 0
Cluster Fragment Queue : 0
Cluster Fragment Discard : 0
Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc bro-2.3.tar.gz)
OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
When I try and configure against my PF_RING libraries, I get this:
./configure --with-pcap=/opt/pfring
Build Directory : build
Source Directory: /root/src/bro-2.3
– The C compiler identification is GNU
– The CXX compiler identification is GNU
– Check for working C compiler: /usr/bin/gcc
– Check for working C compiler: /usr/bin/gcc – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Check for working CXX compiler: /usr/bin/c++
– Check for working CXX compiler: /usr/bin/c++ – works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Found sed: /bin/sed
– Found Perl: /usr/bin/perl
– Found FLEX: 2.5.35
– Found BISON: /usr/bin/bison
– Found PCAP: /opt/pfring/lib/libpcap.so
– Performing Test PCAP_LINKS_SOLO
– Performing Test PCAP_LINKS_SOLO - Failed
– Looking for include files CMAKE_HAVE_PTHREAD_H
– Looking for include files CMAKE_HAVE_PTHREAD_H - found
– Looking for pthread_create in pthreads
– Looking for pthread_create in pthreads - not found
– Looking for pthread_create in pthread
– Looking for pthread_create in pthread - found
– Found Threads: TRUE
– Performing Test PCAP_NEEDS_THREADS
– Performing Test PCAP_NEEDS_THREADS - Failed
CMake Error at cmake/FindPCAP.cmake:61 (message):
Couldn’t determine how to link against libpcap
Call Stack (most recent call first):
cmake/FindRequiredPackage.cmake:26 (find_package)
CMakeLists.txt:52 (FindRequiredPackage)
– Configuring incomplete, errors occurred!
I’m banging my head against this, but I believe this is because bro can’t find the threading library to link to. Could someone point me in the right direction? Do I need other threading libraries? Static linking?
Cheers,
JB
Similar, except I actually use the PF_RING_aware drivers, and transparent mode = 2. So before I perform step 1 I make and make install in PF_RING_aware/non-ZC-drivers/2.6.x/broadcom/netxtreme2-5.2.50/bnx2. Then load the module with modprobe, then I compile PF_RING without issues, and compile tcpdump to work on the new PF_RING. That works fine with tcpdump, but I can’t seem to compile Bro.
Other than that nuance (and the fact that i’m running PF_RING 6.0.2, as mentioned above, not 5.6.2 like the guide) it should be the same.
Any ideas?
Cheers,
JB
I once had an issue with this. My bad workaround was to remove any non PF_RING libpcap. I was noticing some problems with the compiler choosing the wrong one. So using the big hammer, I removed it.
Hrmm...I just tested this now with PF_RING 6.0.1:
Build Directory : build
Source Directory: /home/dev/bro-2.3
-- Found sed: /bin/sed
-- Found Perl: /usr/bin/perl (found version "5.18.2")
-- Found FLEX: 2.5.35
-- Found BISON: /usr/bin/bison
-- Found PCAP: /opt/pfring/lib/libpcap.so
-- Performing Test PCAP_LINKS_SOLO
-- Performing Test PCAP_LINKS_SOLO - Success
-- Looking for pcap_get_pfring_id
-- Looking for pcap_get_pfring_id - found
etc...
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
I didn't see the PCAP_NEEDS_THREADS however. Machine info:
Linux ubuntu-test 3.13.0-34-generic #60-Ubuntu SMP Wed Aug 13 15:49:09 UTC 2014 i686 i686 i686 GNU/Linux
Hope that's at least a little more intel.
James
Doug Burks was quick to point out that i didn’t export LIBS or LDFLAGS.
I would have NEVER guessed this… thanks a thousand times over for this tidbit. Configure finished just fine. Making now. Will update once i’ve got it up and load balanced.
export LDFLAGS="-Wl,–no-as-needed -lrt"
export LIBS="-lrt -lnuma"
Cheers,
JB
So i’ve gone and recompiled with PF_RING 6. I have everything working and using PF_RING correctly, but i’m still seeing packet loss (around 25% on a 400-450mb/s link). I was only ever able to get Bro working with “Transparent mode = 0” and not 2 or 1. I might be doing something completely wrong, but whenever i start BRO, i only ever see one thread peaking at 100%. Here is my node configuration:
[worker-0]
type=worker
host=10.10.10.10
interface=eth3
lb_method=pf_ring
lb_procs=12
Any ideas as to why i’m only getting one thread seeing the bro traffic? Excuse my ignorance.
Cheers,
JB
It's possible that Bro is not actually using PF_RING and is actually
falling back to standard libpcap. Have you checked /proc/net/pf_ring/
to see if there is evidence of Bro using PF_RING?
ldd your bro bin as well:
[09:20:17 ubuntu-test:/usr/local/bro/bin$] ldd bro
linux-gate.so.1 => (0xb778b000)
libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0xb7730000)
James
It sure is. Here is what it’s telling me from the proc fs:
cat /proc/net/pf_ring/53559-eth3.103
Bound Device(s) : eth3
Active : 1
Breed : Non-DNA
Sampling Rate : 1
Capture Direction : RX+TX
Socket Mode : RX+TX
Appl. Name :
IP Defragment : No
BPF Filtering : Enabled
Sw Filt. Rules : 0
Hw Filt. Rules : 0
Poll Pkt Watermark : 1
Num Poll Calls : 1
Channel Id Mask : 0xFFFFFFFF
Cluster Id : 0
Slot Version : 16 [6.0.2]
Min Num Slots : 32768
Bucket Len : 8192
Slot Len : 8232 [bucket+header]
Tot Memory : 269758464
Tot Packets : 220334266
Tot Pkt Lost : 74243221
Tot Insert : 146091045
Tot Read : 145749734
Insert Offset : 136479200
Remove Offset : 136550784
TX: Send Ok : 0
TX: Send Errors : 0
Reflect: Fwd Ok : 0
Reflect: Fwd Errors: 0
Num Free Slots : 0
This is where i’m seeing tons of the packet loss. I’ve got snort running with PF_RING on the same box with 8 threads, 0 packet loss. Any ideas?
Cheers,
JB
Based on the following lines, it looks like Bro is running in standalone mode:
Appl. Name : <unknown>
Cluster Id : 0
If it were running in cluster mode, I would expect to see something
like the following instead:
Appl. Name : bro-eth3
Cluster Id : 21
Have you double-checked your node.cfg?
Have you tried the following?
sudo broctl install && sudo broctl restart
Another thing to check is to search the output of "broctl config"
for "pfringclusterid" (it must be set to a non-zero value if you
want to use PF_RING).
Doug - I fixed my node config up and ran those commands. There were some incorrect configs in the node.cfg file, which i was able to check with the broctl config command.
Everything seems to be working stellar now. Thanks tons for all the help everyone!
Cheers,
JB