A more parallel Bro

Robin Sommer wrote:

real system. So I built a simple tool I call "dns-packet-gen", which

generates

pathological DNS packets constructed for their difficulty in parsing. The

tool

emits pcap traces containing DNS traffic with many records (50+), all of

which

feature deeply-recursive DNS names.

Neat. Is that tool available?

Making it available is on my todo, but it's toward the bottom. My group
currently doesn't have a good web presence, but we're planning to fix that one
of these days as I think it does a lot for a research group's visibility.

I'm wondering how you plan to address cache behavior in your multi-

threaded

implementation of Bro. Obviously, this is a really hard problem, and I'm not
sure how well-studied it is in the literature, especially in the context of IDS.

That's an excellent question and I don't think it has been studied
in the IDS context yet. Our main strategy is to keep related state
together, e.g., by running all analysis involving the same
connection on the same core. It's hard to predict though how cache
effects will eventually turn out to hurt us, and we expect to do a
number of measurements in this regard once the multi-threaded Bro is
in a shape that we can actually measure something. Going from there,
I'm sure there'll be lots of further optimization potential.

I'm going to start analyzing Bro's memory use and performance; do you have
any recommendations about which policy I use? I'm looking for a reasonably
"representative" workload (if such a thing even exists), something that will test
a lot of code paths through the system.

Thanks again,
DA

Here's a sequence of scripts imposing increasing load on Bro that we
have been using in the past for some of our measurements:

(1) tcp
(2) tcp mt
(3) tcp mt scan trw
(4) tcp mt scan trw udp icmp
(5) tcp mt scan trw udp icmp http-request http-reply http-body http-header
(6) tcp mt scan trw udp icmp http-request http-reply http-body http-header ssh pop3 irc ssl smtp

It's also worth comparing running with the connection compressor vs.
without (use_connection_compressor=T/F). And keep an eye on the
packet filter; it's easy to forget but can have quite an impact on
performance depending on what it lets through and what not.

Robin