Seeking hardware/OS recommendations

Zeek
1

Hi,

We are currently running Bro on 2 Dell PowerEdge 2650s.

Each has 2 Syskonnect SK-9844 cards.

Each machine is listening to 2 taps (4 interfaces, which represet ingress
and egress traffic for each of the 2 taps).

The systems are running RedHat Enterprise Linux 4 AS.

From looking at various statistics from the system logs or Bro notice
files, our CPU utilization is generally 99%, and the packet filter seems
to be dropping a high percentage of packets.

We are going to re-design our Bro architecture and are seeking
recommendations for hardware and OS.

We are currently considering running FreeBSD 6.0 instead of RedHat if that
will provide better performance.

We are also considering splitting the collecting and initial log creation
from the subsequent log processing we perform to retain data in our
database. We suspect we will need stronger machines for the initial
collection/log creation than for the subsequent processing, which is
primarily parsing the various log files.

We are looking at Sun Fire X4100 servers with our existing SK-9844 cards
for the "collector" systems. However, it appears that we cannot run
FreeBSD on the X4100 machines due a lack of support for the LSI SAS
(serial attached SCSI) HBA. So, we would instead keep RedHat.

As an alternative, we could use Sun Fire X2100 servers with SK-9E92 cards
for the collectors, running FreeBSD, as long as these would provide
sufficient performance.

We may run 4 collector machines, each listening to its own tap.

We were also thinking of using the Sun Fire X2100s for the secondary log
parsing.

I suppose our questions are:

1) Which OS should we use - FreeBSD or RedHat?

2) Can anyone recommend using the Sun Fire X2100s or X4100s?

3) Does anyone have advice regarding the Syskonnect SK-9844 or SK-9E92 cards?

4) Is it reasonable to assume that the most intensive part of this process
is the initial collection and analysis by Bro which results in the various
Bro log files?

5) Are there other hardware or OS recommendations?

I'm sure I omitting something, but this is a good start.

Thanks in advance for your advice!

Joncarlo Ruggieri
University of CA, Davis
Data Center & Client Services
jruggieri@ucdavis.edu

2

Hi,

We are currently running Bro on 2 Dell PowerEdge 2650s.

Each has 2 Syskonnect SK-9844 cards.

Each machine is listening to 2 taps (4 interfaces, which represet ingress
and egress traffic for each of the 2 taps).

The systems are running RedHat Enterprise Linux 4 AS.

There is a long list of things that can be an issue here:

1) What are the hardware IRQ's for each of the network cards
2) What version of libpcap are you using with bro (an alternative
libpcap from http://public.lanl.gov/cpw/ is known to enhance snort and
other tools. We used it at lanl for our test bro system without load
problems)

The 3 things that we did to heighten performance:

Use a 2.6 kernel with large memory buffers.
Use cards that support this (I do not know about the syscon, but the
eepro1000 worked fine)
Make sure that each network card has a dedicated IRQ to it.

We also found that for most hardware.. we needed to have one dedicated
system per tap for network speeds over 500 mbit/s. The hardware person
thought it was limits on the intel hardware architecture. I left
before we got into studying what changes the AMD hardware architecture
would bring to the mix.

More Answers below:

I suppose our questions are:

1) Which OS should we use - FreeBSD or RedHat?

Whatever floats your boat is my general opinion. I worked for Red Hat
for many years so have a bias towards it.. but I know that the Bro
code seems optimized for FreeBSD.

2) Can anyone recommend using the Sun Fire X2100s or X4100s?

Not I. I have found that the Sun AMD Ultra 20 is really nice, but we
have had problems with the Ultra 40 due to network driver issues.

3) Does anyone have advice regarding the Syskonnect SK-9844 or SK-9E92 cards?

The main things is to find out how they work with NAPI or the
equivalent of NAPI in *BSD kernels.

4) Is it reasonable to assume that the most intensive part of this process
is the initial collection and analysis by Bro which results in the various
Bro log files?

I found that for most packet captures.. the initial capture was as
much work as the analysis. The gurus set it up to use one box for
captures and then regularly pipe over 2 GB files to the analysis
machines. This allowed the analysis to happen with multiple tools and
run as unpriviledged users

5) Are there other hardware or OS recommendations?

I'm sure I omitting something, but this is a good start.

Thanks in advance for your advice!

I think the 2650's might still work with some optimization.. at least
as backup systems until your SunFires (or Alienware or whatever) come
in

3

Bro itself is not really optimized for FreeBSD. Much of its
development happened to take place on FreeBSD but there's hardly any
BSD-specific in it.

Still, in our experience the capture performance of FreeBSD is much
better than that of Linux, at least w/o further Linux kernel patches
(which sometimes show other problems, e.g., not working correctly
with non-blocking the pcap calls used by Bro).

Robin

4

We did some tests with the older Syskonnect SK-9843 V2.0 and we were not happy with the FreeBSD support. Most features of the card could not be used by the official driver. On the other hand there is a brand new driver shipped by marvel for the Yukon chipset of the card. We did not experiment with that yet.

Holger

5

I'll add a couple more points to this thread:

Joncarlo Ruggieri wrote:

Hi,

I suppose our questions are:

1) Which OS should we use - FreeBSD or RedHat?

I'd go with FreeBSD 6.0 too.

Also, its more efficient to combine the interfaces in the kernel
than it is to have Bro listen on 2 interfaces. You can do this with
the 'netgraph' module as follows:

#!/bin/sh
# use NETGRAPH to bond interfaces together

# ti interfaces are real interfaces which receive tap input
# outputs; ngeth0 is created by ngctl

# ng_ether must be loaded so netgraph can "see" the
# real interfaces sf2 and sf3
kldload ng_ether

# bring up the real interfaces
ifconfig ti0 promisc -arp up
ifconfig ti1 promisc -arp up

# create ngeth0 and bind em1 and em2 to it
ngctl mkpeer . eiface hook ether
ngctl mkpeer ngeth0: one2many lower one
ngctl connect ti0: ngeth0:lower lower many0
ngctl connect ti1: ngeth0:lower lower many1

# bring up ngeth0 for sniffing duties
ifconfig ngeth0 -arp up

Also, be sure to check your BPF buffer size (Bro prints this to the info
file on startup) We use 4MB.

4) Is it reasonable to assume that the most intensive part of this process
is the initial collection and analysis by Bro which results in the various
Bro log files?

Some of the analyzers are quite CPU intensive as well. In particular
the HTTP analyzers, and the Signature matching

5) Are there other hardware or OS recommendations?

You should try to keep the CPU load to under 60% to avoid packet drops.

You'll probably need multiple Bro hosts to monitor everything. You can
try doing HTTP on a separate host, or try something like even src/dst
pairs on one host, and odd on another:

eg:
redef restrict_filters += { ["capture even IPs only"] = "(ip[12:4] +
ip[16:4]) & 1 == 0" };