Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))

I'd appreciate feedback on the model for dynamic plugins described
below.

The document contains a short intro on writing a simple plugin, as
well as some background on how/when Bro integrates plugins. The code
is in topic/robin/dynamic-plugins-2.3. It hasn't seen much testing yet
but the basic infrastructure seems to work on Linux and Darwin at
least.

Robin

I'm getting this error:
  error in ./plugins, line 1: read failed with "Is a directory"

It looks like the build/ directory ends up with a directory named "plugins" in it which doesn't play well for when I run Bro in the build directory. :slight_smile:

Anyway, easy to deal with since I can get rid of the plugins directory but maybe something to consider. Maybe just drop a __load__.bro file in there by default?

  .Seth

Doh. :slight_smile: Not sure how to reproducee though. How exactly are you running
it? Are you setting BRO_PLUGIN_PATH, and if so, how?

Robin

I just build bro, cd into the build directory, source in the bro-path-dev.sh script and run Bro.

  .Seth

Couple more observations…

- The init-plugin script is neat. :slight_smile:
- I got the following error when I tried to build my plugin...

[ 40%] Building CXX object CMakeFiles/Bro-LanguageDetect.darwin-x86_64.dir/src/Plugin.cc.o
In file included from /tmp/detect-language/src/Plugin.cc:2:
In file included from /tmp/bro/src/plugin/Plugin.h:9:
In file included from /tmp/bro/src/plugin/Macros.h:11:
In file included from /tmp/bro/src/analyzer/Component.h:6:
In file included from /tmp/bro/src/analyzer/Tag.h:7:
/tmp/bro/src/util.h:24:10: fatal error: 'magic.h' file not found
#include <magic.h>
         ^
1 error generated.
make[3]: *** [CMakeFiles/Bro-LanguageDetect.darwin-x86_64.dir/src/Plugin.cc.o] Error 1
make[2]: *** [CMakeFiles/Bro-LanguageDetect.darwin-x86_64.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [build-it] Error 2

  .Seth

Hi:

Still running through the plugin generation stuff ... seemed to build the binary distribution okay for me after I set BRO (FC 19, x86_64), but haven't tried to really load / run anything yet.

Personally, I'm a fan of this kind of development model. I like not having to worry about synchronizing with master to work on a plugin. I also like that the compilation / link times are much shorter for plugin development than they are when trying to add code to bro directly.

+1 Seth's comment that the init-plugin script is very nice.

A few thoughts:

* Would a section on testing be appropriate? Both btest and unit testing might be useful for plugins.
* A short section explaining how / when to modify CMakeLists.txt might be useful. I had to look at one of the existing analyzer plugins to double-check that all CC files needed to go into bro_plugin_cc(). Not hard, but still might be nice to explicitly document.
* Should plugins be allowed to link to additional libraries? If so, how? I believe this could become an issue if I write a plugin that links against libXYZ, but libXYZ isn't available on the system that's trying to load the plugin.

Cheers,
Gilbert

Ah, I see. It's something else than I thought: a left-over from the
earlier version that isn't needed anymore. Removed.

Robin

I didn't consider Bro's CXX_FLAGS. I think I've fixed that, please try
again.

Robin

* Would a section on testing be appropriate? Both btest and unit
testing might be useful for plugins.

Ack, that's a good point, the init-plugin script could put a basic
setup in place for that, maybe even with a first test making sure
things compile.

* A short section explaining how / when to modify CMakeLists.txt might
be useful.

Yeah, likewise agreed. Indeed tthe documentation needs quite a bit
more material to get people actually started without having to browse
a ton of other code first. I'll leave that for later though once we've
fleshed this all fully out.

* Should plugins be allowed to link to additional libraries?

Yes, definitly. My thinking is that the plugin author will extend the
CMakeIndex.txt with the corresponding pieces, including compile-time
logic to figure out if it's available. However, if the binary module
aims to link against a lib that's not available at runtime where Bro
executes, then I don't think there's much more we can do than fail
loading the plugin: the dlopen will fail (iirc, Bro currently aborts
in that case, I'm not sure if it should proceed without?)

Thanks for the feedback. From chatting with Seth the other day, I took
two more suggestions away:

- I'm coming around that the BRO_PLUGIN_* macros aren't the best way
  of doing things. My main motivation for using them was hiding
  implementation details of the plugin API so that we can more easily
  change things without breaking existing code. However, it seems they
  are putting too much constraints on the plugin writer and/or, if one
  needs to get around them, require a lot of digging into the
  internals. So I'm mulling over creating a (simpler) C++ API to the
  Plugin class that can be used directly.

- The static and dynamic plugins could be unified further. It's
  unclear what the right default is for shipping plugins that provide
  standard functionality, but it would be nice in any case if we could
  just flip a switch to change between static and dynamic builds for
  the in-tree stuff.

Robin

"The static and dynamic plugins could be unified further. It's unclear what the right default is for shipping plugins that provide standard functionality, but it would be nice in any case if we could just flip a switch to change between static and dynamic builds for the in-tree stuff."

What's the reason for supporting both static and dynamic plugin types?

Assuming that eliminating the static plugin type entirely would simplify the build / link process, and that the static plugin type exists largely due to performance concerns: if we could prove that dynamic linking didn't have a significant impact on performance, could we use that as rationale for eliminating static plugin linkage entirely?

Cheers,
Gilbert

That's exactly what I haven't really made up my mind about yet. :slight_smile: I
think there's benefit to having a single Bro binary that comes with
all the standard functionality. One piece is portability: dynamic
linking may not be feasible/possible on some platforms (like tiny
devices, or exotic OSs where our cmake setup may fail to do the right
thing). And I generally like the notion of having just a single binary
with all the standard code included; means less can go wrong (like
version mismatches, etc.)

In terms of performance, I wouldn't be too worried actually, although
it's something that needs testing.

Robin