# Problem: Bro listening on two ethernet interfaces **URL:** \ **Category:** Zeek\ **Created:** [May 10, 2005, 2:38pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747 "2005-05-10T14:38:45Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Vern](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/vern/32/630_2.png) [@Vern](https://community.zeek.org/u/Vern)\ **Post date:** [May 10, 2005, 2:38pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/1 "2005-05-10T14:38:45Z")
> i looked at the c-code. i runned it on different machines and > on various interfaces. bro still drops most of the packets > when i force it to listen on two interfaces. > > is it a libpcap problem? > a bro problem? > a linux problem? I believe it's a Linux problem. We do this under FreeBSD in two different ways, either merging the interfaces in the kernel into one logical interface (via a custom patch), or at user level. While the in-kernel version performs better, the user-level one isn't a disaster like you describe. I also recall hearing others mention that multiple interfaces under Linux do not work well in general. I don't use Linux, though, so can't comment more directly.     Vern ---
**Author:** ![Tim\_Brooks](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Tim\_Brooks](https://community.zeek.org/u/Tim_Brooks)\ **Post date:** [May 10, 2005, 3:01pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/2 "2005-05-10T15:01:29Z")
Ours works fine on linux with the interfaces set in etc/bro.cfg like: BRO\_CAPTURE\_INTERFACE="eth2 eth3" Tim Brooks Vern Paxson wrote: > > i looked at the c-code. i runned it on different machines and > > on various interfaces. bro still drops most of the packets > > when i force it to listen on two interfaces. > > > > is it a libpcap problem? > > a bro problem? > > a linux problem? > > I believe it's a Linux problem. We do this under FreeBSD in two different > ways, either merging the interfaces in the kernel into one logical interface > (via a custom patch), or at user level. While the in-kernel version > performs better, the user-level one isn't a disaster like you describe. > > I also recall hearing others mention that multiple interfaces under Linux > do not work well in general. I don't use Linux, though, so can't comment > more directly. > > Vern > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > Bro mailing list > bro@bro-ids.org > [mailman.icsi.berkeley.edu Mailing Lists](http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro) - -- Tim Brooks Security Engineer National Center for Supercomputing Applications 605 East Springfield Avenue Champaign, IL 61820 ---
**Author:** ![Christoph\_Goeldi](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@Christoph\_Goeldi](https://community.zeek.org/u/Christoph_Goeldi)\ **Post date:** [May 10, 2005, 3:06pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/3 "2005-05-10T15:06:39Z")
hi vern thanks for your statement. > > i looked at the c-code. i runned it on different machines and > > on various interfaces. bro still drops most of the packets > > when i force it to listen on two interfaces. > > > > is it a libpcap problem? > > a bro problem? > > a linux problem? > > I believe it's a Linux problem. We do this under FreeBSD in two different > ways, either merging the interfaces in the kernel into one logical > interface (via a custom patch), or at user level. While the in-kernel > version performs better, the user-level one isn't a disaster like you > describe. i also thought about bonding the interfaces and maybe it won't be a problem. but we plan to introduce Luca Deri's PF\_RING kernel patch in the future and then interface bonding won't be possible any more. > I also recall hearing others mention that multiple interfaces under Linux > do not work well in general. I don't use Linux, though, so can't comment > more directly. do you know other linux software which allows to listen on two interfaces? so that i can use it for testing? tcpdump only allows to listen on one interface or on all (any). thanx christoph ---
**Author:** ![Harry\_Hoffman](https://avatars.discourse-cdn.com/v4/letter/h/6bbea6/32.png) [@Harry\_Hoffman](https://community.zeek.org/u/Harry_Hoffman)\ **Post date:** [May 10, 2005, 5:39pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/4 "2005-05-10T17:39:12Z")
Christoph, If the interfaces don't have an IP address could you bridge them? brctl addbr br0 brctl addif br0 eth0 brctl addif br0 eth1 --Harry Christoph Göldi wrote: > hi vern ... ---
**Author:** ![Christoph\_Goeldi](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@Christoph\_Goeldi](https://community.zeek.org/u/Christoph_Goeldi)\ **Post date:** [May 17, 2005, 11:13am UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/5 "2005-05-17T11:13:16Z")
Hi Tim Zitat von Tim Brooks \: > Ours works fine on linux with the interfaces set in etc/bro.cfg like: > > BRO\_CAPTURE\_INTERFACE="eth2 eth3" Are you realy sure, that Bro doesn't drop the most of the captured packets? I like to know what Linux version (distro), what Bro version and what interfaces (100Mbit or 1Gbit / manufacturer) do you use? Thank you for your time Christoph ---
**Author:** ![Tim\_Brooks](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Tim\_Brooks](https://community.zeek.org/u/Tim_Brooks)\ **Post date:** [May 17, 2005, 7:11pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/6 "2005-05-17T19:11:14Z")
Christoph, I'll let my co-worker, Aashish Sharma, reply to the specific issue of bro dropping packets. However, there are two corrections we made to correct for dropped packets and errors that we were receiving on our 1GB fiber interfaces after first installing bro and turning it on. First, we set the MTU from 1500 to 9000. Second, we set LowLatency=On (i.e. modprobe sk98lin LowLatency=On) That second fix is specific to the fiber cards we are using. After these two changes, we are no longer receiving errors on the interfaces. Aashish Sharma will follow up with bro specific packet loss answer. Thanks, Tim Christoph Goeldi wrote: ---
**Author:** ![Aashish\_Sharma](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@Aashish\_Sharma](https://community.zeek.org/u/Aashish_Sharma)\ **Post date:** [May 19, 2005, 2:21pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/7 "2005-05-19T14:21:52Z")
Hi Chistoph, Tim, All: (I was waiting to get some information/clarifications but nothing yet) Yes, we were/are seeing two very different types of dropped packets notifications : 1) Initially packets were dropped at the interface but as Tim pointed out, that got fixed and current count is substentially low:           RX packets:3550702871 errors:25 dropped:23 overruns:2 frame:2           RX packets:1577193887 errors:13 dropped:13 overruns:0 frame:0 2) Dropped packets notice in the notice.log files. Example: t=1116392591.523558 no=DroppedPackets na=NOTICE\_FILE msg=4475\ packets\ dropped\ after\ filtering,\ 21924\ received Looking at the policy file (netstats.bro) I am inclined to think that these notices are generated because of bro filter. Please correct me here. So in short, since we are using direct network feed, right now, I am relying on error count on interfaces which shows a very low number of packet drops. While with bro we do get dropped packets notices notice.log file which are due to bro filter. What I cannot answer/understand right now is: Is there any way I can find out is bro actually dropping packets, if at all ? Aashish ---
**Author:** ![Christoph\_Goeldi](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@Christoph\_Goeldi](https://community.zeek.org/u/Christoph_Goeldi)\ **Post date:** [May 23, 2005, 4:54pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/8 "2005-05-23T16:54:00Z")
Zitat von Vern Paxson \: > > i looked at the c-code. i runned it on different machines and > > on various interfaces. bro still drops most of the packets > > when i force it to listen on two interfaces. > > > > is it a libpcap problem? > > a bro problem? > > a linux problem? > > I believe it's a Linux problem. We do this under FreeBSD in two different > ways, either merging the interfaces in the kernel into one logical interface > (via a custom patch), or at user level. While the in-kernel version > performs better, the user-level one isn't a disaster like you describe. > > I also recall hearing others mention that multiple interfaces under Linux > do not work well in general. I don't use Linux, though, so can't comment > more directly. I found a small C-program that allows to listen on multiple interfaces and to write the captured packets to a file: [http://www.isi.edu/~hussain/software/snoop.c](http://www.isi.edu/~hussain/software/snoop.c) And it works!!! I'm really not (yet) the pcap-crack. Does somebody know what's the difference between this program and the bro implementation? I really appreciate any help. Cheers Christoph ---
**Author:** ![Jose\_M\_Gonzalez](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@Jose\_M\_Gonzalez](https://community.zeek.org/u/Jose_M_Gonzalez)\ **Post date:** [May 23, 2005, 6:49pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/9 "2005-05-23T18:49:44Z")
Christoph, The C-program you mention opens several interfaces and select()'s its descriptors. A per-packet call to select() can be too expensive in high-volume environments. Moreover, it's not clear select() is the cheapest way to attend several descriptors. If you want to play with this, Kohler's click FromDevice element permits selecting between select(), poll(), and FreeBSD's kevent() (though the latter may be buggy when used with BPF devices). [http://pdos.csail.mit.edu/click/](http://pdos.csail.mit.edu/click/) FYI, Bro tries to limit the calls to select() to just those instants when all the sources are dry (or every often; check IOSource.cc and PktSrc.cc, where all the pcap stuff is located). Also, Bro orders packets received from different sources by their timestamp (the C-program is biased to processing packets from the first interface). BTW, you can't compare this program with Bro. The former just dumps packets to a file. Bro is a stateful intrusion detection system. -Chema Christoph Goeldi wrote: ---
**Author:** ![Christian\_Kreibich3](https://avatars.discourse-cdn.com/v4/letter/c/4af34b/32.png) [@Christian\_Kreibich3](https://community.zeek.org/u/Christian_Kreibich3)\ **Post date:** [May 23, 2005, 7:19pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/10 "2005-05-23T19:19:12Z")
Hi Christoph, > I found a small C-program that allows to listen on multiple interfaces and to > write the captured packets to a file: > [http://www.isi.edu/~hussain/software/snoop.c](http://www.isi.edu/~hussain/software/snoop.c) > > And it works!!! > I'm really not (yet) the pcap-crack. Does somebody know what's the difference > between this program and the bro implementation? I had a quick look at snoop.c and it basically does the most straightforward thing for the task: a select() on the file descriptors associated with the pcap handles of the interfaces. Bro's approach is somewhat more involved because you cannot afford a per-packet select() call on a busy link (see Robin's comments in IOSource.cc). Maybe IOSourceRegistry::FindSoonest() would be a good place to start digging. > I really appreciate any help. I'm sorry I can't help any further regarding this -- if you're on Linux, have you tried letting the kernel sort this out and just use the "any" interface (I forget whether this has been proposed in this thread before)? Cheers, Christian. ---
**Author:** ![Christoph\_Goeldi](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@Christoph\_Goeldi](https://community.zeek.org/u/Christoph_Goeldi)\ **Post date:** [May 23, 2005, 8:28pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/11 "2005-05-23T20:28:10Z")
Hi Chema > The C-program you mention opens several interfaces and select()'s its > descriptors. A per-packet call to select() can be too expensive in > high-volume environments. Moreover, it's not clear select() is the > cheapest way to attend several descriptors. If you want to play with > this, Kohler's click FromDevice element permits selecting between > select(), poll(), and FreeBSD's kevent() (though the latter may be buggy > when used with BPF devices). > > [http://pdos.csail.mit.edu/click/](http://pdos.csail.mit.edu/click/) > > FYI, Bro tries to limit the calls to select() to just those instants > when all the sources are dry (or every often; check IOSource.cc and > PktSrc.cc, where all the pcap stuff is located). Also, Bro orders > packets received from different sources by their timestamp (the > C-program is biased to processing packets from the first interface). Thank you for your explanations. I don't understand the details of these possibilities when capturing packets. But I will try to learn these things to find the problem which appears when bro is listening on multiple interfaces. > BTW, you can't compare this program with Bro. The former just dumps > packets to a file. Bro is a stateful intrusion detection system. I know that this 300-lines-program has not the same functionality like bro! 😉 I just try to understand why the capturing of traffic on multiple interfaces doesn't work with Linux. Thank you for your time Christoph ---
**Author:** ![Christoph\_Goeldi](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@Christoph\_Goeldi](https://community.zeek.org/u/Christoph_Goeldi)\ **Post date:** [May 23, 2005, 8:41pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/12 "2005-05-23T20:41:21Z")
Hi Christian > > I found a small C-program that allows to listen on multiple interfaces > > and to write the captured packets to a file: > > [http://www.isi.edu/~hussain/software/snoop.c](http://www.isi.edu/~hussain/software/snoop.c) > > > > And it works!!! > > I'm really not (yet) the pcap-crack. Does somebody know what's the > > difference between this program and the bro implementation? > > I had a quick look at snoop.c and it basically does the most > straightforward thing for the task: a select() on the file descriptors > associated with the pcap handles of the interfaces. > > Bro's approach is somewhat more involved because you cannot afford a > per-packet select() call on a busy link (see Robin's comments in > IOSource.cc). Maybe IOSourceRegistry::FindSoonest() would be a good > place to start digging. Okay. I'll try to figure out more about this ominous select(). > > I really appreciate any help. > > I'm sorry I can't help any further regarding this -- if you're on Linux, > have you tried letting the kernel sort this out and just use the "any" > interface (I forget whether this has been proposed in this thread > before)? I'll try the any interface tomorrow. But it wouldn't solve my problems anyway because I want to specifically select the observed interfaces and not capture the packets of all interfaces of this host. Thanks for your help. Cheers Christoph ---
**Author:** ![Jose\_M\_Gonzalez](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@Jose\_M\_Gonzalez](https://community.zeek.org/u/Jose_M_Gonzalez)\ **Post date:** [May 24, 2005, 12:45am UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/13 "2005-05-24T00:45:12Z")
Christoph Göldi wrote: > \> BTW, you can't compare this program with Bro. The former just dumps > \> packets to a file. Bro is a stateful intrusion detection system. > > I know that this 300-lines-program has not the same functionality > like bro! 😉 > I just try to understand why the capturing of traffic on multiple > interfaces doesn't work with Linux. Slightly offtopic, I had a related problem capturing traffic in Linux (RHEL WS 4 running 2.6.9-5.ELsmp), in just one interface at the same time. I tried injecting packets into a network device, and capturing them in the same machine (tcpreplay/tcpdump). In both cases, I used a 230 Mbps, real-traffic trace that lasts 3.4 seconds (~250000 packets). When I used a real device (Intel PRO/100 Ethernet Driver), tcpreplay got slowed down to 90 Mbps, and tcpdump got duplicated packets (not that many, ~262 out of 250000). When I used a virtual device (VMWare's vmnet), tcpdump lost ~66 consecutive packets out of the 250000, which were reported as "dropped by kernel." Maybe somebody has some ideas. -Chema ---
**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\ **Post date:** [May 24, 2005, 7:31am UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/14 "2005-05-24T07:31:48Z")
As a general note, the IOSource code is still rather new and may not yet perform optimal in all cases. The major problem is that the precise semantics of the libpcap interface varies depending on OS version and libpcap version (and patches to either of them). I would not be suprised if the problem w/ two Linux interfaces is related to this. I'd be happy to help sorting this out. However, I don't have a Linux machine at hand which has two interfaces seeing a high-volume stream (and I am a bit tight on time right now...). Robin ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:37pm UTC](https://community.zeek.org/t/problem-bro-listening-on-two-ethernet-interfaces/747/15 "2022-05-06T15:37:27Z")