I'm wondering if we should add another type of plugin component:
threads. This would be for functionality that's to run in parallel
with Bro's main thread, and communicate with it via message passing.
We have the structure for that in place already, logging and reading
are using it as already. But this would formalize the notion a bit
more directly that a plugin can provide a new thread, with its own
logic; and also extend the interface that it has available from inside
that thread (e.g., being able to raise events; have bif functions
that, when called, get passed through to the thread).
One use case is Christian's OpenFlow plugin: if we went the route of
integrating an external library for speaking OpenFlow directly, that
communication needs to be handled somewhere. Traditionally, that would
be an IOSource hooked into the main loop. The plugin model could
support that, too, but being able to fully decouple it inside its own
thread seems appealing.
Jon, how are you planing to integrate Broker into Bro? Would this help
there as well if you could just follow a similar structure with Broker
running inside its own thread?
Very timely question, I've be mulling over this and I would like to vote for adding thread component.
This may allow us to do a lot more processing of data in the script land.
Now my use case may not be likely an ideal one.
I am *experimenting* with a policy to flag very long sustained persistant connections between two hosts (irrespective of ports). For this, I am storing connection information [src, dst] in a table and based on various conditions/heuristics I am expiring these at different dynamic times (so read_expire isn't helpful). For 'spike' elimination from scanners, I iterate through the table every min. (unoptimal but stay with me). At present when table stores about 4 Million elements, iterating through it freezes entire bro (no growth of conn log etc).
A seperate thread could allow me to delegate these kinds of tasks outside Bro's main thread/event queue.
Well, that's a different thing. I was talking about plugins in core
land, written in C++. Threading in script-land is nothing I see
happening soon unfortunately ...
Mostly planning to integrate as IOSource(s) in the main thread as I expect many of the messages either originate or terminate there due to script-land interactivity. Broker’s internal processing is to be scheduled to threads automatically by actor-framework. Wrapping all Broker-related messaging as a Bro-thread that does its own message passing seems like an unneeded indirection as Broker already facilitates its own message passing between endpoints (as it’s built on top of the more-generalized actor-framework message passing).
Would it make sense to replace the existing inter-thread communication code with the broker / porting the existing writers and readers to use the actor framework? This way, there would only be a single, shared message-passing mechanism inside of bro, instead of having one for core <-> threads, and another for core <-> external
I think Broker is going to end up best addressing "core <-> external” by providing (hopefully) simple interfaces to a few common/specific messaging patterns and may not make sense to use it generally for “core <-> threads”. But yes, using actor-framework directly to replace the current threading approach is an option to consider.
