# Bro 2.0 packets dropped **URL:** \ **Category:** Zeek\ **Created:** [February 3, 2012, 1:38pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208 "2012-02-03T13:38:09Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Machiel\_van\_Veen](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@Machiel\_van\_Veen](https://community.zeek.org/u/Machiel_van_Veen)\ **Post date:** [February 3, 2012, 1:38pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/1 "2012-02-03T13:38:09Z")
Hello, I am getting a lot of messages reporting dropped packages. This is on CentOS 6.2 running 3 workers on a quad core machine with 8 gb of ram, sas disks. The link I monitor has low traffic, 0.2 k packets p/s avg, max 2.0 k peaks, the server load is very low. I would expect no packets to be dropped. Besides tuning the receive buffer and queue length is there anything else I can do about this? worker-1: 1328274953.996680 recvd=129059158 dropped=114860 link=129174018 worker-2: 1328274954.197859 recvd=129059218 dropped=115120 link=129174338 worker-3: 1328274954.397642 recvd=129052866 dropped=122170 link=129175036 Thank you, Machiel. ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [February 3, 2012, 1:52pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/2 "2012-02-03T13:52:11Z")
Are you monitoring 3 separate links on three interfaces? I'm a little suspicious that you may be monitoring the same traffic three separate times. You will need to load balance the traffic across those three workers if it's a single interface (I'm working on automating this now). Could you add a line to load the misc/capture-loss script to your local.bro? @load misc/capture-loss After you do that, make sure you do "check", "install", "restart" in broctl. The capture-loss script will give you another measure of packet loss that is not based on information being received from the NIC. Oh, that brings up another question. What NICs are you using?   .Seth ---
**Author:** ![Machiel\_van\_Veen](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@Machiel\_van\_Veen](https://community.zeek.org/u/Machiel_van_Veen)\ **Post date:** [February 3, 2012, 3:18pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/3 "2012-02-03T15:18:29Z")
> \> Besides tuning the receive buffer and queue length is there anything else > \> I can do about this? > \> > \> worker-1: 1328274953.996680 recvd=129059158 dropped=114860 link=129174018 > \> worker-2: 1328274954.197859 recvd=129059218 dropped=115120 link=129174338 > \> worker-3: 1328274954.397642 recvd=129052866 dropped=122170 link=129175036 > > Are you monitoring 3 separate links on three interfaces? I'm a little > suspicious that you may be monitoring the same traffic three separate > times. You will need to load balance the traffic across those three > workers if it's a single interface (I'm working on automating this now). It is one interface, there might be a problem load balancing. I've switched to a standalone setup for now. "listening on eth1, capture length 8192 bytes" "bro: 1328281729.277621 recvd=3553337 dropped=4503 link=3557842" The packetloss is still there though. > Could you add a line to load the misc/capture-loss script to your > local.bro? @load misc/capture-loss > > After you do that, make sure you do "check", "install", "restart" in > broctl. The capture-loss script will give you another measure of packet > loss that is not based on information being received from the NIC. > From the alarm summary: "2012-02-03-15:39:46 CaptureLoss::Too\_Much\_Loss The capture loss script detected an estimated loss rate above 27.282%" > Oh, that brings up another question. What NICs are you using? Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz driver: bnx2 version: 2.1.11 firmware-version: bc 4.6.0 ipms 1.6.0 >   .Seth Thanks again, Machiel. ---
**Author:** ![William\_Jones](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@William\_Jones](https://community.zeek.org/u/William_Jones)\ **Post date:** [February 5, 2012, 6:53pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/4 "2012-02-05T18:53:49Z")
Are you using pf\_ring? ---
**Author:** ![William\_Jones](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@William\_Jones](https://community.zeek.org/u/William_Jones)\ **Post date:** [February 5, 2012, 8:03pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/5 "2012-02-05T20:03:20Z")
It looks like its 10. ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [February 6, 2012, 6:55pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/6 "2012-02-06T18:55:26Z")
> It is one interface, there might be a problem load balancing. I've switched to > a standalone setup for now. If you aren't taking any steps to load balance the traffic then it definitely isn't working. We don't have automated load balanced configuration available in BroControl yet. 🙂 Today, I did just write a script that automates a BPF based load balancing technique on clusters which will be getting merged in along with the rest of the automated load balancing code soon. > "bro: 1328281729.277621 recvd=3553337 dropped=4503 link=3557842" > "2012-02-03-15:39:46 CaptureLoss::Too\_Much\_Loss > The capture loss script detected an estimated loss rate above 27.282%" Are sniffing from a tap or a SPAN port? I'm a little suspicious because the first line indicates that the NIC was showing 0.1% packet loss, but the second line indicates much more loss. The misc/capture-loss.bro script can detect loss due to reasons beyond the monitoring host (like an overloaded SPAN port) so I'm just trying to figure out where there is a such a huge disparity between the two measurements. Oh, one other thought. Are you disabling all of the offload features of your NIC? Here's an article about it:   1. [Security Onion: When is full packet capture NOT full packet capture?](http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html) Is the MTU on your NIC larger than 8192 (Bro 2.0's default snaplen). If there are packets larger than that they won't be seen by default. > > Oh, that brings up another question. What NICs are you using? > > Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz > driver: bnx2 > version: 2.1.11 > firmware-version: bc 4.6.0 ipms 1.6.0 I usually recommend not using Broadcom nics for monitoring. At times with various broadcom nics I've run into weird problems so I tend to avoid them.   .Seth ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [February 6, 2012, 6:56pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/7 "2012-02-06T18:56:44Z")
I'm not sure what you're asking. There is no hard coded number of workers that can be defined for a cluster. We have deployments of anywhere from 2 workers up to over a hundred.   .Seth ---
**Author:** ![William\_Jones](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@William\_Jones](https://community.zeek.org/u/William_Jones)\ **Post date:** [February 6, 2012, 7:30pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/8 "2012-02-06T19:30:22Z")
Here my problem. I have a single server and a defined 10 works on it to divided up the load. Here the output of "broctl status" root@homey manager]# broctl status Name Type Host Status Pid Peers Started manager manager homey.tacc.utexas.edu running 6202 11 05 Feb 15:22:15 proxy-1 proxy homey.tacc.utexas.edu running 6237 11 05 Feb 15:22:17 worker-1 worker mojo1.tacc.utexas.edu running 18356 2 05 Feb 15:26:41 worker-10 worker mojo1.tacc.utexas.edu running 18350 2 05 Feb 15:26:41 worker-2 worker mojo1.tacc.utexas.edu running 18348 2 05 Feb 15:26:41 worker-3 worker mojo1.tacc.utexas.edu running 18349 2 05 Feb 15:26:41 worker-4 worker mojo1.tacc.utexas.edu running 18357 2 05 Feb 15:26:41 worker-5 worker mojo1.tacc.utexas.edu running 18352 2 05 Feb 15:26:41 worker-6 worker mojo1.tacc.utexas.edu running 18353 2 05 Feb 15:26:41 worker-7 worker mojo1.tacc.utexas.edu running 18354 2 05 Feb 15:26:41 worker-8 worker mojo1.tacc.utexas.edu running 18355 2 05 Feb 15:26:41 worker-9 worker mojo1.tacc.utexas.edu running 18351 2 05 Feb 15:26:41 I now add woker-11 to to the configuration and "bro status" returns: BroControl] \> status Name Type Host Status Pid Peers Started manager manager homey.tacc.utexas.edu running 29316 12 06 Feb 13:16:59 proxy-1 proxy homey.tacc.utexas.edu running 29351 12 06 Feb 13:17:01 worker-1 worker mojo1.tacc.utexas.edu running 25026 2 06 Feb 13:17:06 worker-10 worker mojo1.tacc.utexas.edu running 25028 2 06 Feb 13:17:06 worker-11 worker mojo1.tacc.utexas.edu running 25033 2 06 Feb 13:17:06 worker-2 worker mojo1.tacc.utexas.edu running 25032 2 06 Feb 13:17:06 worker-3 worker mojo1.tacc.utexas.edu running 25025 2 06 Feb 13:17:06 worker-4 worker mojo1.tacc.utexas.edu running 25031 2 06 Feb 13:17:06 worker-5 worker mojo1.tacc.utexas.edu running 25029 2 06 Feb 13:17:06 worker-6 worker mojo1.tacc.utexas.edu running 25027 2 06 Feb 13:17:06 worker-7 worker mojo1.tacc.utexas.edu running 25034 2 06 Feb 13:17:06 worker-8 worker mojo1.tacc.utexas.edu running 25030 2 06 Feb 13:17:06 worker-9 worker mojo1.tacc.utexas.edu running 25036 ??? 06 Feb 13:17:06 Notice the ???. It an indication that something is not working correct;y the bro communication library. ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [February 6, 2012, 7:46pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/9 "2012-02-06T19:46:02Z")
If you run "ps.bro" in broctl, what do you get? I'm suspicious that the old #9 process didn't die and it's still holding the communication port open which would result in that error.   .Seth ---
**Author:** ![Machiel\_van\_Veen](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@Machiel\_van\_Veen](https://community.zeek.org/u/Machiel_van_Veen)\ **Post date:** [February 9, 2012, 7:41am UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/10 "2012-02-09T07:41:25Z")
Sorry for replying so late, I was unexpectedly off-line for some time... I tried all the suggestions yesterday and found the issue to be the Broadcom NIC. The tap is on a Intel NIC now and dropped packages are at an acceptable level. bro: 1328772549.657147 recvd=22987443 dropped=9811 link=22997255 There have been no alerts regarding packet loss either. Thanks for you support on this issue, back to clustering for me 😉 ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [February 9, 2012, 1:55pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/11 "2012-02-09T13:55:07Z")
Great to hear! Broadcom nics have the weirdest problems. Are you doing network based clustering (multiple physical hosts)?   .Seth ---
**Author:** ![sridhar\_basam1](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@sridhar\_basam1](https://community.zeek.org/u/sridhar_basam1)\ **Post date:** [February 9, 2012, 3:23pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/12 "2012-02-09T15:23:02Z")
Not exactly related to Machiel problem but if people are stuck with Broadcom adapters. You can try updating the firmware on the cards. The version above 4.x is extremely old. [http://www.broadcom.com/support/ethernet\_nic/netxtremeii.php](http://www.broadcom.com/support/ethernet_nic/netxtremeii.php) for getting software for your netextreme-ii adapters. You can also increase the ring buffer from the default 255 to 1024 or more so that your receive ring isn't overflowing. "/sbin/ethtool -G eth0 rx 1024" The same would apply for intel adapters too, if you notice your interface card dropping packets.       sridhar ---
**Author:** ![Machiel\_van\_Veen](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@Machiel\_van\_Veen](https://community.zeek.org/u/Machiel_van_Veen)\ **Post date:** [February 10, 2012, 8:12am UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/13 "2012-02-10T08:12:07Z")
I'm looking to run one physical host for now and add a second with several workers later. However load balancing has not worked yet, the module and driver are loaded and pfcount shows traffic but Bro reports everything triple, no load balancing. Is there any configuration apart from configuring the manager, proxy and workers in node.cfg done in Bro to get this working? As I understood there should not be, as long as you link to the pf\_ring enabled libpcap it should work when pf\_ring is active. Machiel. ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [February 10, 2012, 2:25pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/14 "2012-02-10T14:25:04Z")
Could you send me the content of your node.cfg and broctl.cfg files? This is fortunate timing, I've been preparing a blog post about using PF\_Ring load balancing with Bro and it would be good to find out if there are any problems with it.   .Seth ---
**Author:** ![Machiel\_van\_Veen](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@Machiel\_van\_Veen](https://community.zeek.org/u/Machiel_van_Veen)\ **Post date:** [February 10, 2012, 2:49pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/15 "2012-02-10T14:49:47Z")
I'm using the following in node.cfg: [manager] type=manager host=10.0.0.42 [proxy-1] type=proxy host=10.0.0.42 [worker-1] type=worker host=10.0.0.42 interface=p1p1 [worker-2] type=worker host=10.0.0.42 interface=p1p1 [worker-3] type=worker host=10.0.0.42 interface=p1p1 And these settings in broctl.cfg: MailTo = root@localhost SitePolicyStandalone = local.bro SpoolDir = /var/opt/bro/spool LogDir = /var/opt/bro/logs LogRotationInterval = 3600 MinDiskSpace = 5 Debug = 1 Regards, Machiel. ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [February 10, 2012, 3:26pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/16 "2012-02-10T15:26:48Z")
> > > Is there any configuration apart from configuring the manager, proxy and > > > workers in node.cfg done in Bro to get this working? > > > > Could you send me the content of your node.cfg and broctl.cfg files? This > > is fortunate timing, I've been preparing a blog post about using PF\_Ring > > load balancing with Bro and it would be good to find out if there are any > > problems with it. > > > > .Seth > > I'm using the following in node.cfg: That seems fine. > And these settings in broctl.cfg: > > MailTo = root@localhost > SitePolicyStandalone = local.bro > SpoolDir = /var/opt/bro/spool > LogDir = /var/opt/bro/logs > LogRotationInterval = 3600 > MinDiskSpace = 5 > Debug = 1 It looks like you are missing the setting that turns on the pf\_ring clustering support. If you built against the pf\_ring libpcap wrapper it should have been put in there automatically (unless you installed over top of a previous installation?). Add this to your broctl.cfg and do "check", "install", "restart" in broctl. PFRingClusterId = 21   .Seth ---
**Author:** ![Machiel\_van\_Veen](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@Machiel\_van\_Veen](https://community.zeek.org/u/Machiel_van_Veen)\ **Post date:** [February 10, 2012, 3:46pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/17 "2012-02-10T15:46:40Z")
I've added the option, there is no difference. I did notice in the debug logs before that this option has been set by default. At startup i see the following for all workers, proxy and manager: "PCAP\_PF\_RING\_USE\_CLUSTER\_PER\_FLOW=1 PCAP\_PF\_RING\_CLUSTER\_ID=21" The bro binary does seem to use the correct lib: $ ldd /opt/bro/bin/bro | grep pcap libpcap.so.1 =\> /usr/local/lib/libpcap.so.1 (0x00007fae5cad2000) I'll go ahead and do this again on monday, perhaps I did make a mistake during the build process. Thanks, Machiel. ---
**Author:** ![Martin\_Holste](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@Martin\_Holste](https://community.zeek.org/u/Martin_Holste)\ **Post date:** [February 10, 2012, 9:42pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/18 "2012-02-10T21:42:43Z")
What do you see in /proc/net/pf\_ring/ ? If you cat a file matching the PID of one of the Bro processes, it should say what the cluster\_id is. If they are all 21, then it is working. ---
**Author:** ![William\_Jones](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@William\_Jones](https://community.zeek.org/u/William_Jones)\ **Post date:** [February 12, 2012, 3:38pm UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/19 "2012-02-12T15:38:45Z")
There is a relative new behavior from the scanners. In order are to work around the automatic scan blocking they have increased the scan rate to so that they can scan 30K-60K address in a second. This make bro go compute bound, I think it do to creating a recorded for each connection pair, and it cannot keep up. Using PF\_RING helps but not all attach hash well and one worker can be be overwhelmed. Has anyone else seeing this new behavior. Bill Jones ---
**Author:** ![Machiel\_van\_Veen](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@Machiel\_van\_Veen](https://community.zeek.org/u/Machiel_van_Veen)\ **Post date:** [February 13, 2012, 9:30am UTC](https://community.zeek.org/t/bro-2-0-packets-dropped/2208/20 "2012-02-13T09:30:50Z")
It looks like only one worker uses pfring, the clusterid is 21. $ ls -l /proc/net/pf\_ring/ r--r--r-- 1 root root 0 Feb 13 10:15 15489-p1p1.1 dr-xr-xr-x 5 root root 0 Feb 13 10:15 dev -r--r--r-- 1 root root 0 Feb 13 10:15 info -r--r--r-- 1 root root 0 Feb 13 10:15 plugins\_info [BroControl] \> status ... worker-1 worker 192.168.42.215 running 15489 2 13 Feb 10:12:02 ... When I set transparent\_mode to 2 it shows: [BroControl] \> netstats   worker-1: 1329124697.533460 recvd=144655 dropped=0 link=144655   worker-2: 1329124697.733532 recvd=0 dropped=0 link=0   worker-3: 1329124697.934520 recvd=0 dropped=0 link=0 The other two workers do not connect. The only thing I could find so far which could cause this is quick\_mode, I've disabled this option. Any idea what else could cause this? Machiel. [Next page](https://community.zeek.org/t/bro-2-0-packets-dropped/2208.md?page=2)