Improving Bro's main loop

Development
1

Just starting a discussion to take inventory of the current problems with Bro’s main loop and ideas for how to improve it. Let’s begin with a list of issues (please comment if you have additions):

(1) It uses select(), which is the worst polling mechanism. It has an upper limit on number of fds that can be polled (some OSs are fixed at 1024), and also scales poorly. This is problematic for Bro clusters that have many nodes/peers.

2) Integrating new I/O sources isn’t always straightforward from a technical standpoint (e.g. see [1]). I also found that it’s difficult to understand the ramifications of any change to the run loop without also digging into esoteric details you may not initially think are related (e.g. I usually had to double-check the internals of I/O or threading systems when making any change to the main loop, which may mean there's basic problems with those abstractions).

3) Bro’s time/timers are coupled with I/O. Time does not move forward unless there is an active I/O source. This isn’t usually a functional problem for users, but devs occasionally have to hack around it (e.g. unit tests).

I think CAF [2] and/or libuv [3] can address these issues:

1) libuv: abstracts whatever polling mechanism is best for the OS you’re on. CAF: could allow a more direct actor messaging interface to Broker and since remote communication takes the bulk of fds being polled, the remaining fds (e.g. packet sources, etc.) could be fine to poll in whatever fashion, while the remote communication then is subject to CAF’s own multiplexer.

2) Both libuv and CAF use abstractions/models that are shown to work well. I think the actor model, by design, does a better job of encouraging systems that are decoupled and therefore scalable.

3) Both libuv and CAF have mechanisms that could implement timers into the run loop such that they’d work independently of other I/O.

libuv may be a quicker, more straightforward path to fixing (1), which is the most critical issue, but it’s also the easiest to fix without aid of a library. Libuv can also replace other misc. code in Bro like async DNS and signal handling, but, while those may be crufty, they aren’t frequent sources of pain.

Since CAF is a requirement of Broker already and has most potential to improve/replace parts of Bro’s threading system and the way in which Broker is integrated, it may be best in the long-term to explore moving things out of Bro’s current run loop by making them into actors that use message-passing interfaces and then relying on CAF’s own loop.

Any thoughts?

- Jon

[1] http://mailman.icsi.berkeley.edu/pipermail/bro-dev/2015-May/010069.html
[2] https://actor-framework.org/
[3] http://docs.libuv.org/en/v1.x/

2

Nice summary, I agree with all of the pain points. Without thinking
much about solutions yet, a bit of random brainstorming on things to
keep in mind when thinking about this:

    - We need to maintain some predictability in scheduling, in
      particular with regarding to timing/timers. Bro's network time
      time is, by definition, defined through I/O. My gut feeling is
      that we need to keep the tight coupling there, as otherwise
      semantics would change quite a bit.

    - Related, another reason for time playing such an important role
      in the I/O loop is that Bro needs to process its soonest input
      first. That's most important for packet sources: if we have
      packets coming from multiple packet sources, earlier timestamps
      must be processed before later ones across all of them.

    - Time is generally complex, we have three different notions of
      network time actually, all with some different specifics: time
      during real-time processing, time during offline trace
      processing, and pseudo-realtime.

    - I believe we need to maintain the ability to have I/O loops that
      don't have FDs.

    - I like the idea of using CAF, including because it's going to be
      a required dependency anyways in the future. I would also like
      it conceptually to move I/O to actors, and I'm wondering if even
      packets sources could go there. However, I can't quite tell if
      that's feasible given other constrains and how other parts of
      the system are layed out (including that in the end, everything
      needs to go back into the main thread before being further
      processed; at least for the time being).

    - One of the trickiest parts in the past has been ensuring good
      performance on a variety of platforms and OS versions. Whatever
      we do, it'll be important to do quite a bit of test-driving and
      benchmarking. Let's try to structure the work so that we can get
      to a prototype quickly that allows for some initial performance
      validation of the approach taken.

Robin

3

   - We need to maintain some predictability in scheduling, in
     particular with regarding to timing/timers. Bro's network time
     time is, by definition, defined through I/O. My gut feeling is
     that we need to keep the tight coupling there, as otherwise
     semantics would change quite a bit.

   - Related, another reason for time playing such an important role
     in the I/O loop is that Bro needs to process its soonest input
     first. That's most important for packet sources: if we have
     packets coming from multiple packet sources, earlier timestamps
     must be processed before later ones across all of them.

   - Time is generally complex, we have three different notions of
     network time actually, all with some different specifics: time
     during real-time processing, time during offline trace
     processing, and pseudo-realtime.

Also not sure to what degree coupling related to time/timers can be reduced, though I think at least an initial refactor of the run loop could be done such that it doesn’t change much related to how time currently works. Then maybe later or during the refactor, it will get easier to see what exactly can be improved.

   - I believe we need to maintain the ability to have I/O loops that
     don't have FDs.

Yep, don’t think there will be a problem there.

   - I like the idea of using CAF, including because it's going to be
     a required dependency anyways in the future. I would also like
     it conceptually to move I/O to actors, and I'm wondering if even
     packets sources could go there. However, I can't quite tell if
     that's feasible given other constrains and how other parts of
     the system are layed out (including that in the end, everything
     needs to go back into the main thread before being further
     processed; at least for the time being).

I do think even packet sources could get moved into actors. My initial idea for the main loop refactor is for it to be a single actor waiting for “ready for processing” messages from IOSources, and then for each IOSource to be responsible for its own FD polling (if it needs it). That way, the main loop doesn’t care about FDs at all anymore and if an IOSource needs to poll FDs it can just use poll() in its own actor/thread for now (my guess is that most IOSources will just have a single FD to poll anyway or that the polling mechanism isn’t a very significant chunk of time for ones that may have more, but the only way to answer that is to actually do the performance testing.)

   - One of the trickiest parts in the past has been ensuring good
     performance on a variety of platforms and OS versions. Whatever
     we do, it'll be important to do quite a bit of test-driving and
     benchmarking. Let's try to structure the work so that we can get
     to a prototype quickly that allows for some initial performance
     validation of the approach taken.

Sure. I was also expecting to try and just get something working without any significant overhauling of any of Bro’s systems.

- Jon

4

Yeah, one basic decision we'll have to make is how much logic to move
into threads. Conceptually, that's the right thing to do, but we need
to make sure the code is thread-safe, and it generally makes
development and debugging harder in the future. CAF helps with all of
that, but all the legacy code worries me in that regard. That said,
the IOSources are pretty much self-contained and probably not very
problematic in that way. (But then: having some code needing to be
thread-safe, while other parts break every rule in the book in that
regard, is also confusing; we have that challenge already with the
logging and input frameworks.))

Robin