Throughput Problems

Zeek
1

Hi All,

I have been testing Bro recently. I have been having some performance issues. I can load these any and all (without the bad ones listed below) of these policies I get great performance:

@load site
@load alarm
@load weird
@load http
@load worm
@load blaster
@load hot
@load signatures
@load synflood
@load backdoor

If I add any single one of these I go from being able to process traffic at 90Mb/s to under 1 Mb/s.

@load login
@load irc
@load portmapper
@load http-request
@load http-reply
@load ftp
@load stepping
@load tftp
@load frag
@load smtp

Has anyone ever seen this problem before? Know the solution? Know to where even start looking?

I was also curious at waht speeds people start dropping packets. Obviously the traffic your monitoring has an impact so maybe a little background would help too. (ie 100 Mb/s with 64k udp packets)

Thanks in advance,

// Joel

Joel Ebrahimi

2

What OS are you running under? First immediate guess is that your
BPF buffers are much too small.

    Vern

3

I am running using Suse on PowerPC. I am also using specialty hardware from Bivio.
I do not belive it is an issue with BPF.

// Joel

4

I also just tested this on a dual core intel platform running FC 6.
I get the same exact behavior. In this case I did the following:

./configure
make
make install
make install-brolite

( i had to remove the pre-generated parser code as well)

When I remove loading brolite from the site file I get exceptional performance.

With brolite enabled it almost appears as if bro does nothing. When I send a test pcap with 30K packets bro is only able to proces 1800. The cpu usage of bro never even gets passed 5%.

// Joel

5

I am running using Suse on PowerPC. I am also using specialty hardware from
Bivio.
I do not belive it is an issue with BPF.

Well, it's very likely *some* issue with packet capture, since I believe
the difference between your policy-scripts-that-work and scripts-that-don't
is that the latter capture full-sized packets and the former basically don't.

Try this. Run with the set of scripts that work plus print-filter.bro to
see what filter is being used. Then run with the scripts that don't work
plus print-filter and get that filter. See then how tcpdump fares using
each filter (along with -s 0 to capture full-sized packets).

If that doesn't shed light, then what are the dominant types of appications
in your traffic, and how do you fare using Bro setups that don't capture them?

We routinely run on traffic with 100+ Mbps traffic (18K pps), predominantly
SSH and HTTP, without significant problems with drops.

    Vern

6

One thing to quickly try is to tune the packet capture on
Linux, as the defaults are pretty bad.

  See:
http://www.net.t-labs.tu-berlin.de/research/hppc/

  For the definitive guide.

The shortcut version is, as root:

echo 33554432 > /proc/sys/net/core/rmem_default
echo 33554432 > /proc/sys/net/core/rmem_max
echo 10000 > /proc/sys/net/core/netdev_max_backlog

  which will temporarily increase the size of the packte capture
buffers to 32 MB, and see if that makes a difference.

7

Vern,

Thanks for this info. I had not realized that brolite was applying a number of filters. My main testing pcap was a large variety of services,protocols, and sessions. My testing was semi-automated to check packets sent vs packets bro received. The fillter scewed my results. I applied a redef to the packet filter and I am now seeing excellent statistics on the intel machine.

I will use this information now to re-test on the Bivio platform.

// Joel