Shipping CAF with Broker?

Development
1

I was wondering the other day if we could add CAF as submodule to
Broker, and then just start compiling it along with everything else. A
long time ago we decided generally against shipping dependencies along
with Bro, but in this case it might make people's lives quite a bit
easier, as hardly any Bro user will have CAF installed already. And
even if they had (say from an earlier Bro version), it might not be
the right version. If we included it with Broker, things would just
work.

We could even go a step further and compile CAF statically into
libbroker, so that in the end from a user's perspective all they deal
with is Broker: if they link against it, they get everything they
need.

Would that make sense?

Robin

2

I think we're quite a ways off from CAF being generally packaged with most operating systems (especially having the correct version to work with broker). I think that including it with broker and building libcaf into libbroker statically makes sense too. I've been concerned about the difficulty of building Bro from source too.

   .Seth

3

This will also make

    pip install pybroker

feasible, which would be a huge usability improvement for integrating external systems with bro.

4

I agree - while I see the argument of not packaging external stuff, I think we should make an exception here.

Johanna

5

Sounds fine to me. Also means one less variable (CAF version) to get
under control when troubleshooting/debugging reported issues.

- Jon

6

Sounds like everybody likes this idea. Jon, want to take a stab at it?
Seems like something we should do before merging the branch into
master so that we get everybody gets on the right track right away.

Let's try the the static library approach: link CAF statically into
libbroker. I'm not 100% sure if that's straight-forward to do, but I
hope so ...

Robin

7

I can understand where you’re coming from. Dependency management in C++ is lackluster, to say the least. However, baking CAF into Broker in this way is not trivial and can break many, many things.

CAF headers are included in public broker headers. This means users of libbroker must be able to find those headers in the system paths. You could install CAF headers via CMake along the Broker headers, but that would override any other local CAF installation. Worse, if a user installs CAF from a different source afterwards you could produce an ABI clash when compiling a libbroker application with the wrong CAF headers. It might even compile fine, as long as the API remains compatible, but produce very nasty hard-to-debug errors at runtime.

Moving *all* CAF headers out of Broker headers could solve this, but it requires a lot of pimpl boilerplate code to make everything CAF opaque. Completely hiding away the CAF actor system also takes away any opportunity for integrations. For example, we are currently thinking about how we could integrate Broker with VAST. Locking away the actor system would take away many benefits of integrating Broker. For one, we would have to run two distinct actor systems in the same process instead of using one and getting all the scalability benefits from it. Two actor systems means two schedulers that either constantly fight for resources or leave performance on the table. If each scheduler gets only half of the available HW via config that would waste a lot of CPU bandwidth if one scheduler is idle but the other at capacity. Performance aside, not having access to the Broker actor system would potentially require us to duplicate a lot of code for using Broker types in VAST. Of course this is also prone to error now, because any change in Broker no longer automatically updates the serialization code in the now detached actor system.

Long story short, doing this would shut a lot of doors for us. Did you consider using something like CMake’s External Project [1] feature or Conan [2]? There is a Conan recipe for CAF (contributed by a user a while ago) that could make life for Bro users easier. I’m happy to work on the recipe if it isn’t working out of the box right now, if that would work for you as an alternative.

    Dominik

[1] https://cmake.org/cmake/help/v3.0/module/ExternalProject.html
[2] https://www.conan.io/

8

CAF headers are included in public broker headers.

Good point, I didn't remember that, it does complicate the situation.

Though maybe it's still only more a problem for the less common
use-case of someone actually wanting to use libbroker themselves. I
think more commonly someone just wants to get Bro up and running and
so there could possibly get away with static linking and not bothering
to install headers.

Moving *all* CAF headers out of Broker headers could solve this, but it requires a lot of pimpl boilerplate code

That does sound like a bunch of work someone probably would not enjoy doing :slight_smile:

Though Broker originally used to pimpl everything and that approach
worked and seemed fine at the time.

Completely hiding away the CAF actor system also takes away any opportunity for integrations.

Yeah, that all sounds less desirable and not sure what could be done about it.

[1] ExternalProject — CMake 3.0.2 Documentation

Maybe as a last resort. Generally, I think I'd rather not -- every
time I've tried to use them or seen them used there's been a
"flakiness" to them that I wish I could describe in more logical
terms. I also think every project where I've tried to use a CMake
External Project myself has never actually made it into an actual
release or at least any that are still used at all. I'm not sure
there's a relation, though it also doesn't make me feel great about
the prospect of using that same approach here.

[2] https://www.conan.io/

Could be something to consider/evaluate; never used it myself.

At this point, I'd rather just stick to the current situation until
it's actually clear what problems people will have (or take the time
to complain about), if any. Then once pain points are better known,
decide on what would improve the situation.

- Jon

9

Yeah, same here, I didn't think about that part either, it's
definitely a concern. Not immediately sure if there's a middle-ground
that would give us the best of both worlds: easy of installation for
Bro users, while remaining flexible for external Broker/CAF usages.
But agree that this needs more thought before moving ahead with
anything. Thanks for bringing that up, Dominik.

Robin