Calling external functions in binpac protocol parser

Hi All,

(1) I wonder that what’s the rationales of removing the binpac files for some common protocols (e.g., HTTP, DNS, et al.)? Does current bro distribution only include the handwritten protocol parsers for those protocols?

I can find the http-{protocol, analyzer}.pac files have been removed since bro-2.2. I checked the CHANGE log but cannot find the explanation.

(2) We create a “general” analytic module that includes APIs (e.g., passing a key/value pair) can be called by multiple protocol parsers such as HTTP and DNS (essentially we only want the “parser” instead of the whole “analyzer” part; that’s the reason we are looking for the http-protocol.pac).

We develop such module as a plugin, say “Sample::Test” which includes a function test_func(…). We have another sample protocol parser including following code:

type myPDU() = record {
data: bytestring &restofflow;
} &byteorder = bigendian & let{
deliver: bool = $context.flow.myFUNC();
};

flow myFLOW() {
flowunit = myPDU(…);

function myFUNC() {
Sample::test_func(…);
}

That is, in current sample module we want the external function being called when receiving a protocol flow PDU (in &let {…}). So how we can get the binpac (protocol parser) recognize the function Sample::test_func() written in another plugin Sample::Test? I can see in /src/analyzer/protocols, the analyzers can include the functionality from another analyzer by including the headers such as #include “analyzer/protocols/tcp/…”. But when writing the plugins in /aux/bro-aux/plugin-support, how can we do that?

Thanks very much!

Hi Shuai,

(1) I wonder that what's the rationales of removing the binpac files for
some common protocols (e.g., HTTP, DNS, et al.)? Does current bro
distribution only include the handwritten protocol parsers for those
protocols?

The protocol parsers that were removed were incomplete and not used by
Bro. For example for HTTP, which you also mention, the binpac analyzer was
never enabled AFAIK.

I can find the http-{protocol, analyzer}.pac files have been removed since
bro-2.2. I checked the CHANGE log but cannot find the explanation.

I assume that since this is not a user-visible change (as I said they
never were used) back in those days we decided this did not warrant a
CHANGE entry :).

If you are still interested into them, you can just get them from the bro
2.2 tarball, there was no further development of them. But - as I said -
note that they never were actually used.

(2) We create a "general" analytic module that includes APIs (e.g., passing
a key/value pair) can be called by multiple protocol parsers such as HTTP
and DNS (essentially we only want the "parser" instead of the whole
"analyzer" part; that's the reason we are looking for the
http-protocol.pac).

I assume you are aware that the -parser and -analyzer in binpac is just a
naming convention and all that the -analyzer is is the part that has all
the callbacks into Bro that raises events?

We develop such module as a plugin, say "Sample::Test" which includes a
function test_func(...). We have another sample protocol parser including
following code:

> type myPDU() = record {
> data: bytestring &restofflow;
> } &byteorder = bigendian & let{
> deliver: bool = $context.flow.myFUNC();
>};

> flow myFLOW() {
> flowunit = myPDU(...);
>
> function myFUNC() {
> Sample::test_func(...);
> }

That is, in current sample module we want the external function being
called when receiving a protocol flow PDU (in &let {...}). So how we can
get the binpac (protocol parser) recognize the function Sample::test_func()
written in another plugin Sample::Test? I can see in
/src/analyzer/protocols, the analyzers can include the functionality from
another analyzer by including the headers such as #include
"analyzer/protocols/tcp/...". But when writing the plugins in
/aux/bro-aux/plugin-support, how can we do that?

Ok, I will give a slightly long answer to this. First - assuming that the
test function is just a c++ function, chances are that you want to develop
it outside of the binpac files - you can e.g. move it into a separate
class that is accessible by everyone. Depending on the things that your
function does, it even might be possible to make it a static function.

The second answer is - since binpac files are only compiles to c++ you
have to do the same thing that the other protocol analyzers do - include
the headers using #include statements. I think you can just use relative
paths from where the files are located, in addition to absolute paths from
the bro base. So - doing something like #include "../your.h" might work
fine.

Also note that you probably should not put your plugins into
bro-aux/plugin-support in the first case. Having them in a separate
directory is probably preferable - completely outside of the Bro source
tree.

I hope this helps a bit,
Johanna

Thanks for your detailed reply, Johanna!

Hi again,

> you can e.g. move it into a separate
> class that is accessible by everyone. Depending on the things that your
> function does, it even might be possible to make it a static function.
>

Yes, as you mentioned, we do want a module that can be accessed and called
by every protocol analyzer. Regarding the "separate class that is
accessible by everyone",
where we should implement this module? Do we need put the C++ files in
bro/src?

Thinking about it again - this actually is a good question. It would be
neat if you could add functionality to your plugins without having to
modify the base Bro installation (which is what you basically do when you
move things to /bro/src and include them into the compile process there -
which certainly is a theoretical possibility).

But - it would be nicer if that was not necessary. So I think my first
idea would be to just put the shared sources into a directory that is
accessible by all your plugins, and compile it together with a plugin (in
separate namespaces so that it does not conflict afterwards).

This should be fine assuming that your shared functionality is not ultra
huge..

> Also note that you probably should not put your plugins into
> bro-aux/plugin-support in the first case. Having them in a separate
> directory is probably preferable - completely outside of the Bro source
> tree.
>

Sorry I don't get the point of this part. In my understanding, wherever we
develop
the plugins, it will be complied into bro's library /bro/lib/.../plugin.
What do you mean
(and how) "outside of the Bro source tree"?

Yes, your plugins will reside inside the Bro installation in the end.
However you probably should keep your sourecode separate from the main Bro
sourcecode - that's all what I meant :).

Johanna