I'm trying to make Bro run multi-threaded... so wanted to ask if you
guys have any experience with that. First of all, is Bro written to be
multi-threaded? If not, can you guys point me to the places which
would need some work to make it multi-threaded. Apparently with the
complex interplay of data structures, it is hard to find those. I have
fixed one such place in Type.cc in:
BroType* base_type(TypeTag tag)
where it was using a static base_types array. There may more lurking
elsewhere, which I haven't been able to find.
Esp, I'm facing this issue, where I'm trying to create a new Val()
object and bro coredumps in one of the threads.
I'm trying to make Bro run multi-threaded... so wanted to ask if you
guys have any experience with that. First of all, is Bro written to be
multi-threaded?
Not currently, no.
If not, can you guys point me to the places which
would need some work to make it multi-threaded. Apparently with the
complex interplay of data structures, it is hard to find those. I have
fixed one such place in Type.cc in:
BroType* base_type(TypeTag tag)
where it was using a static base_types array. There may more lurking
elsewhere, which I haven't been able to find.
Esp, I'm facing this issue, where I'm trying to create a new Val()
object and bro coredumps in one of the threads.
There are surely many places in the code in which static variables may
collide with multi-threaded operation. There will also be a number of
synchronization issues. The real question is what you are actually
trying to achieve, so you can adjust the architecture to run the
relevant tasks in separate threads. This may be non-trivial. That said,
it would clearly be interesting to parallelize the operation of
individual analyzers, for example.
I'm playing with the idea of remotely controlling Bro
operation/configuration. Sorry, I'm not interested in Broccoli as that
is a non-standard interface. For that I'm creating a separate thread
that accepts SOAP messages and controls Bro on-the-fly. I could really
do it as a single thread, but it's cleaner the other way.
I'm curious though, if the broccoli interface/api is a binary or a text one!
Your idea of parallelizing the various analyzers is something I have
considered. Really, it'll only make sense parallelizing the analyzers
on a single packet. Analyzing multiple packets at the same instant
would create synchronization problems of their own sort. What if the
analysis of second packet ends up finishing before the first though it
was dependent on the first? This is true not just for packets from
same connection. Distinct connections may be related too and may need
synchronous processing.
Talking of analyzing single packets (at an instant) using multiple
analyzers, don't know how beneficial that is really. Can't think of
too many cases where this is helpful. Especially, even in these cases,
after analyzing the first packet, mostly only a single analyzer
remains interested. And from what I know, subsequent packets belonging
to the same connection can re-use the analyzer information stored in
the connection entry.
Having said that, parallelizing Event dispatches would be interesting,
since there's no guarantee of order in Event Handler execution for a
given event anyways.
Sorry, I'm not interested in Broccoli as that
is a non-standard interface. For that I'm creating a separate thread
that accepts SOAP messages and controls Bro on-the-fly.
Well sorry likewise, as we won't be interested in your possible contribution
in that case.
Your idea of parallelizing the various analyzers is something I have
considered. Really, it'll only make sense parallelizing the analyzers
on a single packet.
Please see our papers which explore the possibilities in depth:
Rethinking Hardware Support for Network Analysis and
Intrusion Prevention,
V. Paxson et al., Proc. USENIX Hot Security, August 2006.
An Architecture for Exploiting Multi-Core Processors to
Parallelize Network Intrusion Prevention,
V. Paxson, R. Sommer, and N. Weaver,
Proc. IEEE Sarnoff Symposium, May 2007, to appear.
> Sorry, I'm not interested in Broccoli as that
> is a non-standard interface. For that I'm creating a separate thread
> that accepts SOAP messages and controls Bro on-the-fly.
Well sorry likewise, as we won't be interested in your possible contribution
in that case.
I hope my comment didn't come across as forthright sacrilege, or a
snide advice for that matter. I just meant that requirements differ in
my case. I trust that Broccoli does well at what it is inteded to.
Please see our papers which explore the possibilities in depth: