Can someone please comment on the current status of multi-threading in
Bro?
That will need to be Robin, as he's the one who's done all the work on
this. However, he's on vacation for another week, and will no doubt face
a major email backlog when he returns.
Can someone please comment on the current status of multi-threading in
Bro? I would be interested in doing some work here.
We have a proof-of-concept implementation of a multi-threaded Bro.
Even though still an early prototype, it already improves Bro's
performance quite a bit on multi-core systems and demonstrates that
the approach works quite well. However, this prototype still has a
number of limitations and is not yet usable from an operational
perspective. There are also a number of different routes we could go
from here, which aren't fully clear yet in their specifics.
For more background, the most current description of the prototype
is here:
Section V. describes the parallelization approach, and Section VI.
presents some preliminary measurements. (Section I-IV are on a more
conceptual level; not all of that is directly reflected in Bro).
A limiting factor for moving this forward right now is available
time, so help and contributions would certainly be welcome. Is there
anything specific you're thinking about? (I saw your mail about
GPUs, will reply to that in a bit).
Thanks for sharing the link to the paper, it made an interesting read. This paper does a great job of explaining the concepts involved, even for someone like myself who doesn't have a background in parallel computing.
Clearly, an IDS architecture that separates protocol analysis and event handling can employ this technique to improve performance. And so this can be used for Bro. But, you'd need a working ANI.
I don't know how recently this paper was written, but when we're talking about today, where does ANI fit in, in hardware, and if not implemented as custom hardware then as a small program running in a core " if a multicore fabric includes embedded network resources" (like UltraSPARC T2)?
I couldn't figure out how recently this paper was written (2007-08?), and so while reading this paper I couldn't help but think about this very basic question-
Today, if I'm using Bro as the Host-based IDS on my machine, and if I find that Bro is not being able to keep up with the incoming packet rate, what are some steps that I should take?
I'm guessing you meant network based IDS (as opposed to Host-based)?
Currently, if you are trying to scale Bro as a network IDS the most viable method is to use the cluster deployment using the BroControl utility. It's currently being used in production at a number of locations. For more documentation about BroControl and the cluster deployment you can refer to the following link.
Clearly, an IDS architecture that separates protocol analysis and event
handling can employ this technique to improve performance. And so this
can be used for Bro. But, you'd need a working ANI.
That's right, but note that the ANI in the paper is a more powerful
component than what we need for "just" parallelizing a passive NIDS
(such as Bro). The latter primarily needs a load-balancer that
distributes packets across threads in a predictable manner. In the
most simple implemention (and in the current prototype) that's just
another thread copying packets around, which is obviously not that
great. A number of things come to mind to improve on that (as you
already mention as well): an external load-balancer like what we use
for the Bro Cluster; some decicated network processers can already
do this internally; and, probably the best option of all, some of
the new commodity NICs actually have the necessary functionality on
board and can steer traffic directly to their target threads.
Generally, I expect much of what we need here to become pretty much
standard functionality in the near future.
I don't know how recently this paper was written,
The paper has been growing over a while. The later parts were
finished about a year ago, the earlier ones in 2007/8 alreday iirc.
The later parts were