i looked at the c-code. i runned it on different machines and
on various interfaces. bro still drops most of the packets
when i force it to listen on two interfaces.
is it a libpcap problem?
a bro problem?
a linux problem?
I believe it's a Linux problem. We do this under FreeBSD in two different
ways, either merging the interfaces in the kernel into one logical interface
(via a custom patch), or at user level. While the in-kernel version
performs better, the user-level one isn't a disaster like you describe.
I also recall hearing others mention that multiple interfaces under Linux
do not work well in general. I don't use Linux, though, so can't comment
more directly.
Ours works fine on linux with the interfaces set in etc/bro.cfg like:
BRO_CAPTURE_INTERFACE="eth2 eth3"
Tim Brooks
Vern Paxson wrote:
i looked at the c-code. i runned it on different machines and
on various interfaces. bro still drops most of the packets
when i force it to listen on two interfaces.
is it a libpcap problem?
a bro problem?
a linux problem?
I believe it's a Linux problem. We do this under FreeBSD in two different
ways, either merging the interfaces in the kernel into one logical interface
(via a custom patch), or at user level. While the in-kernel version
performs better, the user-level one isn't a disaster like you describe.
I also recall hearing others mention that multiple interfaces under Linux
do not work well in general. I don't use Linux, though, so can't comment
more directly.
i looked at the c-code. i runned it on different machines and
on various interfaces. bro still drops most of the packets
when i force it to listen on two interfaces.
is it a libpcap problem?
a bro problem?
a linux problem?
I believe it's a Linux problem. We do this under FreeBSD in two different
ways, either merging the interfaces in the kernel into one logical
interface (via a custom patch), or at user level. While the in-kernel
version performs better, the user-level one isn't a disaster like you
describe.
i also thought about bonding the interfaces and maybe it won't be a problem.
but we plan to introduce Luca Deri's PF_RING kernel patch in the future and
then interface bonding won't be possible any more.
I also recall hearing others mention that multiple interfaces under Linux
do not work well in general. I don't use Linux, though, so can't comment
more directly.
do you know other linux software which allows to listen on two interfaces?
so that i can use it for testing?
tcpdump only allows to listen on one interface or on all (any).
I'll let my co-worker, Aashish Sharma, reply to the specific issue of
bro dropping packets. However, there are two corrections we made to
correct for dropped packets and errors that we were receiving on our
1GB fiber interfaces after first installing bro and turning it on.
First, we set the MTU from 1500 to 9000.
Second, we set LowLatency=On (i.e. modprobe sk98lin LowLatency=On)
That second fix is specific to the fiber cards we are using. After
these two changes, we are no longer receiving errors on the interfaces.
Aashish Sharma will follow up with bro specific packet loss answer.
2) Dropped packets notice in the notice.log files. Example:
t=1116392591.523558 no=DroppedPackets na=NOTICE_FILE msg=4475\ packets\ dropped\ after\ filtering,\ 21924\ received
Looking at the policy file (netstats.bro) I am inclined to think that these notices are generated because of bro filter. Please correct me here.
So in short, since we are using direct network feed, right now, I am relying on error count on interfaces which shows a very low number of packet drops. While with bro we do get dropped packets notices notice.log file which are due to bro filter.
What I cannot answer/understand right now is:
Is there any way I can find out is bro actually dropping packets, if at all ?
i looked at the c-code. i runned it on different machines and
on various interfaces. bro still drops most of the packets
when i force it to listen on two interfaces.
is it a libpcap problem?
a bro problem?
a linux problem?
I believe it's a Linux problem. We do this under FreeBSD in two different
ways, either merging the interfaces in the kernel into one logical interface
(via a custom patch), or at user level. While the in-kernel version
performs better, the user-level one isn't a disaster like you describe.
I also recall hearing others mention that multiple interfaces under Linux
do not work well in general. I don't use Linux, though, so can't comment
more directly.
The C-program you mention opens several interfaces and select()'s its
descriptors. A per-packet call to select() can be too expensive in
high-volume environments. Moreover, it's not clear select() is the
cheapest way to attend several descriptors. If you want to play with
this, Kohler's click FromDevice element permits selecting between
select(), poll(), and FreeBSD's kevent() (though the latter may be buggy
when used with BPF devices).
FYI, Bro tries to limit the calls to select() to just those instants
when all the sources are dry (or every often; check IOSource.cc and
PktSrc.cc, where all the pcap stuff is located). Also, Bro orders
packets received from different sources by their timestamp (the
C-program is biased to processing packets from the first interface).
BTW, you can't compare this program with Bro. The former just dumps
packets to a file. Bro is a stateful intrusion detection system.
And it works!!!
I'm really not (yet) the pcap-crack. Does somebody know what's the difference
between this program and the bro implementation?
I had a quick look at snoop.c and it basically does the most
straightforward thing for the task: a select() on the file descriptors
associated with the pcap handles of the interfaces.
Bro's approach is somewhat more involved because you cannot afford a
per-packet select() call on a busy link (see Robin's comments in
IOSource.cc). Maybe IOSourceRegistry::FindSoonest() would be a good
place to start digging.
I really appreciate any help.
I'm sorry I can't help any further regarding this -- if you're on Linux,
have you tried letting the kernel sort this out and just use the "any"
interface (I forget whether this has been proposed in this thread
before)?
The C-program you mention opens several interfaces and select()'s its
descriptors. A per-packet call to select() can be too expensive in
high-volume environments. Moreover, it's not clear select() is the
cheapest way to attend several descriptors. If you want to play with
this, Kohler's click FromDevice element permits selecting between
select(), poll(), and FreeBSD's kevent() (though the latter may be buggy
when used with BPF devices).
FYI, Bro tries to limit the calls to select() to just those instants
when all the sources are dry (or every often; check IOSource.cc and
PktSrc.cc, where all the pcap stuff is located). Also, Bro orders
packets received from different sources by their timestamp (the
C-program is biased to processing packets from the first interface).
Thank you for your explanations. I don't understand the details of
these possibilities when capturing packets. But I will try to learn
these things to find the problem which appears when bro is listening
on multiple interfaces.
BTW, you can't compare this program with Bro. The former just dumps
packets to a file. Bro is a stateful intrusion detection system.
I know that this 300-lines-program has not the same functionality
like bro!
I just try to understand why the capturing of traffic on multiple
interfaces doesn't work with Linux.
And it works!!!
I'm really not (yet) the pcap-crack. Does somebody know what's the
difference between this program and the bro implementation?
I had a quick look at snoop.c and it basically does the most
straightforward thing for the task: a select() on the file descriptors
associated with the pcap handles of the interfaces.
Bro's approach is somewhat more involved because you cannot afford a
per-packet select() call on a busy link (see Robin's comments in
IOSource.cc). Maybe IOSourceRegistry::FindSoonest() would be a good
place to start digging.
Okay. I'll try to figure out more about this ominous select().
I really appreciate any help.
I'm sorry I can't help any further regarding this -- if you're on Linux,
have you tried letting the kernel sort this out and just use the "any"
interface (I forget whether this has been proposed in this thread
before)?
I'll try the any interface tomorrow. But it wouldn't solve my problems
anyway because I want to specifically select the observed interfaces and
not capture the packets of all interfaces of this host.
> BTW, you can't compare this program with Bro. The former just dumps
> packets to a file. Bro is a stateful intrusion detection system.
I know that this 300-lines-program has not the same functionality
like bro!
I just try to understand why the capturing of traffic on multiple
interfaces doesn't work with Linux.
Slightly offtopic, I had a related problem capturing traffic in Linux
(RHEL WS 4 running 2.6.9-5.ELsmp), in just one interface at the same
time.
I tried injecting packets into a network device, and capturing them in
the same machine (tcpreplay/tcpdump). In both cases, I used a 230 Mbps,
real-traffic trace that lasts 3.4 seconds (~250000 packets).
When I used a real device (Intel PRO/100 Ethernet Driver), tcpreplay got
slowed down to 90 Mbps, and tcpdump got duplicated packets (not that many,
~262 out of 250000).
When I used a virtual device (VMWare's vmnet), tcpdump lost ~66 consecutive
packets out of the 250000, which were reported as "dropped by kernel."
As a general note, the IOSource code is still rather new and may not
yet perform optimal in all cases. The major problem is that the
precise semantics of the libpcap interface varies depending on OS
version and libpcap version (and patches to either of them). I would
not be suprised if the problem w/ two Linux interfaces is related to
this.
I'd be happy to help sorting this out. However, I don't have a Linux
machine at hand which has two interfaces seeing a high-volume stream
(and I am a bit tight on time right now...).
