Cmake for Bro

Development
1

(Moving this to bro-dev, let's keep dev discussions there).

I can see that happening. I'll plan for that to happen unless we
find some other CMake projects that are doing something more
brilliant.

Ok, I think this is my favorite option so far, it will also solve
the --help problem.

1) --enable-shippedpcap, "use the shipped version of libpcap"

Please get rid of that, we don't want to ship a pcap anymore at all.

2) bison vs. byacc

I think the main reason for supporting both was that some systems
used to come with the former, and others with the latter. I'm not
sure how that's today, but I'd be fine supporting just bison. Vern,
what do you think?

Just looked at the other (Bro-specific) configure options, below are
my thoughts. Anybody, please check whether you agree (active mapping
and DFA cache expiration I'm most unsure about).

Robin

--------- cut -------------------------------------------------------

  --enable-brov6 enable IPV6 processing
    -> Keep; we should eventually turn this always on and remove the
    configure option, but we aren't quire there yet.

  --enable-int64 enable use of int64 (long long) for integers
    -> Remove; let's always turns this on and remove the configure option.
  
  --enable-activemapping enable active mapping processing
    -> Do we want to keep active mapping in the code? Is anybody
    using that?
    
  --enable-expire-dfa-states enable DFA state expiration
    -> Do we want to keep this in code? Nobody is probably using
    it, and it complicates the DFA code quite a bit.
  
  --enable-debug no compiler optimizations
    -> Keep.
  
  --disable-select-loop disable select-based main loop
    -> Remove; no longer needed.
  
  --enable-perftools use Google's perftools
    -> Keep.
  
  --enable-shippedpcap use the shipped version of libpcap
    -> Remove.
  
  --disable-nbdns Disable non-blocking DNS support
    -> Remove; don't think anybody uses this.
  
  --disable-largefile omit support for large files
    -> Remove.
  
  --disable-broccoli Do not build/package Broccoli
    -> Remove; broccoli will be its own package.
  
  --disable-broctl Do not build/package broctl framework
    -> Remove; BroControl will be its own package.
    
  --enable-cluster Configure broctl for cluster usage
    -> Remove; that's an option for BroControl.

  --with-openssl=PATH path to OpenSSL (needed for SSL analyzer and secure communication)
    -> Keep, but I'm wondering whether we should make OpenSSL a
    mandatory dependency to get rid of all the "#if openssl"
    conditions.
  
  --with-perl=PATH path/name of the Perl interpreter
     -> Remove; I don't think we're using Perl anywhere anymore.
  
  --with-dag=PATH path to the DAG library (for native support for Endace Tech.'s DAG monitoring cards)
     -> Remove; nobody has been using the native DAG support in
        ages.
  
  --with-binpac=PATH path to a binpac executable for compiling analyzer code
     -> Keep.

2

--enable-activemapping enable active mapping processing
   -> Do we want to keep active mapping in the code? Is anybody
   using that?

What is active mapping? I've seen that for a long time but never asked what it is.

--disable-nbdns Disable non-blocking DNS support
   -> Remove; don't think anybody uses this.

Isn't the nonblocking dns support used for lookups in "when" blocks?

--with-openssl=PATH path to OpenSSL (needed for SSL analyzer and secure communication)
   -> Keep, but I'm wondering whether we should make OpenSSL a
   mandatory dependency to get rid of all the "#if openssl"
   conditions.

I vote for making it a mandatory dependency. At the very least, it's another assumption we can make when developing scripts (that SSL support exists) and is there anything that doesn't include OpenSSL in the base distribution?

  .Seth

3

I think the main reason for supporting both was that some systems
used to come with the former, and others with the latter.

That's my memory too.

I'm not
sure how that's today, but I'd be fine supporting just bison. Vern,
what do you think?

Let's give that a try.

In general, I agree with your config suggestions. Some comments:

  --enable-activemapping enable active mapping processing
    -> Do we want to keep active mapping in the code? Is anybody
    using that?

I don't believe anyone uses this, so it makes sense to remove it.
(Seth: this is an alternative to in-line normalization for resisting
evasion attacks. It works by building up a map of how different end
systems resolve ambiguities. You can read more about it if you want at
http://www.icir.org/vern/papers/activemap-oak03.pdf .)

  --enable-expire-dfa-states enable DFA state expiration
    -> Do we want to keep this in code? Nobody is probably using
    it, and it complicates the DFA code quite a bit.

Seems okay to remove it.

  --disable-nbdns Disable non-blocking DNS support
    -> Remove; don't think anybody uses this.

We can get the equivalent via setenv BRO_DNS_FAKE, right? (Seth: this is
to *avoid* using non-blocking DNS because it won't locally build.)

  --with-openssl=PATH path to OpenSSL (needed for SSL analyzer and secure communication)
    -> Keep, but I'm wondering whether we should make OpenSSL a
    mandatory dependency to get rid of all the "#if openssl"
    conditions.

I like that notion. I guess we'll see whether it leads to portability
problems, similar to bison.

  --with-dag=PATH path to the DAG library (for native support for Endace Tech.'s DAG monitoring cards)
     -> Remove; nobody has been using the native DAG support in
        ages.

Are we pretty sure about that?

    Vern

4

--with-perl=PATH path/name of the Perl interpreter
-> Remove; I don't think we're using Perl anywhere anymore.

So far there are two places I found dependent on Perl:

1) bro/src/make_parser.pl

Because we decided to require bison over byacc, this can be removed since in the bison case it just does:

    cp $srcdir/parse.in $builddir/parse.y"

2) bro/src/make_dbg_constants.pl

I didn't see an immediate way out of this one. Do you have thoughts on if something can be done with it?

- Jon

5

2) bro/src/make_dbg_constants.pl

I didn't see an immediate way out of this one. Do you have thoughts on
if something can be done with it?

I would think it's not hard to rewrite in another scripting language,
if we decide to standardize on say Python. It's just generating a bunch
of header files from the info in DebugCmdInfoConstants.in . This is used
for Bro's interactive debugger, which has fallen into disrepair - but I don't
think we should just stop supporting the debugger entirely, as it's potentially
really quite handy functionality.

    Vern

6

Do you have a sense how much work it will take to get it back into
shape?

Robin

7

I don't think I've ever heard from anybody (except me :slight_smile: using it.
People are using Endace's libpcap wrapper though.

Let me ask on the mailing list.

Robin

8

I'm definitely interested in how broken it is. I vaguely remember using it at one point a while ago, but maybe I didn't use the broken parts?

  .Seth

9

First a general question: for the configure options we decide to remove, are we talking about just removing them from the user-facing configure script or additionally cleaning out the preprocessor blocks from the code that implements them?

--disable-broccoli Do not build/package Broccoli
-> Remove; broccoli will be its own package.

--disable-broctl Do not build/package broctl framework
-> Remove; BroControl will be its own package.

--enable-cluster Configure broctl for cluster usage
-> Remove; that's an option for BroControl.

I'm not fully understanding how the new packaging and build system are going to cooperate.

It seems like this will make each sub-module's build independent from the main Bro "core". That would be fine (and probably easiest thing to implement), but I'm not sure that's what we want?

I thought that conflicts with the goal of providing an easy method of retrieve/configure/build/install for all desired sub-modules in one shot. In order to do that, the top-level Bro configure script needs to be able to enable sub-modules and be aware of their config options.

CMake is fine with recursing on sub-projects and git can do sub-modules, so I was expecting to leverage that functionality.

- Jon

10

for the configure options we decide to remove,
are we talking about just removing them from the user-facing configure
script or additionally cleaning out the preprocessor blocks from the code
that implements them?

I was thinking the latter. We can dig the goop out of the revision archives
if we decide to resurrect one of them.

I thought that conflicts with the goal of providing an easy method of
retrieve/configure/build/install for all desired sub-modules in one shot.

Well, I'm wondering about this too. But it sounds like other folks have
a model in mind whereby this isn't much of a hassle.

    Vern

11

2) bison vs. byacc

I think the main reason for supporting both was that some systems
used to come with the former, and others with the latter. I'm not
sure how that's today, but I'd be fine supporting just bison. Vern,
what do you think?

IIRC, somebody also suggested to ship Bro with the generated scanner and
parser so that a default build wouldn't need to run flex/bison/yacc.
Don't know whether that's a good idea though.

cu
Gregor

12

I agree. I think using the libpcap wrapper is the way to got. That's
also what the TimeMachine uses.

cu
Gregor

13

I'm not fully understanding how the new packaging and build system are going to cooperate.

Frankly, it's not totally clear to me yet either ... My current
assumption was that all the packages are first of all independent,
and need to be downloaded/configured/installed separately. Then, we
could go from there and see how we can make things easier for people
working from sources, for example by providing a meta-distribution
bro-all which comes with all parts and has some glue/magic to make
their joint installation easier. But not sure how that would look
like, suggestions welcome.

CMake is fine with recursing on sub-projects and git can do sub-modules, so I was expecting to leverage that functionality.

If we can make this work, I'd be happy to do so. How would this look
like, say for passing options to the sub-project's configure? And
how would building separate packages for each module work then?

Robin

14

Me too.

Robin

15

> CMake is fine with recursing on sub-projects and git can do
> sub-modules, so I was expecting to leverage that functionality.

If we can make this work, I'd be happy to do so. How would this look
like, say for passing options to the sub-project's configure?

Take git out of the equation for now and say we downloaded the 'bro-all' source bundle containing the Bro super-project and all sub-projects. The super-project's configure wrapper will have to be able to recursively look for additional options provided by any existing sub-projects, but only has to display and maybe validate them. Setting a sub-project option via the super-project's configure wrapper will end up as a globally scoped CMake variable although only maybe a single sub-project cares about it.

If you consider how this works with git sub-modules, the only thing that's different is how you obtain the source bundle. Here, cloning the Bro super-project will give you empty sub-module directories that one can init/update as desired:

    git submodule init <path>
    git submodule update <path>

The super-project just maintains sub-module pointers to a particular commit in their repository (probably a release tag).

how would building separate packages for each module work then?

That stumped me. After reading around a bit, it seems like CPack does not currently play nice with sub-projects (although it's on someone's TODO list).

So it looks like we can't just do a `make package` at the super-project level and expect it to spit out source & binary packages for independent sub-projects. But if each module as a standalone works fine with CPack, I don't see why we can't just make a package generating script that iterates over modules, doing an independent out-of-source build and package creation for each.

The 'bro-all' package should still be possible via CPack if sub-projects had some extra logic to disable their own CPack settings when they detect that they're part of a super-project.

Thoughts on those approaches?

- Jon

16

I like this, sounds like it should work.

The super-project's configure wrapper will have to be able to
recursively look for additional options provided by any existing
sub-projects, but only has to display and maybe validate them.

The wrapper should also make clear which options apply to which
sub-package so that one doesn't get confused. Generally, I expect
the number of options to remain small though, so this shouldn't be
much of a problem.

the Bro super-project will give you empty sub-module directories
that one can init/update as desired:

    git submodule init <path>
    git submodule update <path>

Sounds like we could provide a script which does this for each
submodule.

standalone works fine with CPack, I don't see why we can't just make
a package generating script that iterates over modules, doing an
independent out-of-source build and package creation for each.

That sounds fine.

The 'bro-all' package should still be possible via CPack if

I don't think we necessarily need a bro-all package. I see bro-all
mainly as for those folks building/developing from source.

Robin

17

One more thing: broctl has three further aux/* sub-modules:
capstats, pysubnettree, and trace-summary.

These are actually all usable without broctl so I think we should
make them separate modules as well (I'm already maintaining them in
their own Subversion repositories, which so far has been a pain
because I had to apply all changes both to these repos and to the
Bro repository. Once in git as separate modules, I'll switch over to
using just those).

Robin

18

> for Bro's interactive debugger, which has fallen into disrepair

Do you have a sense how much work it will take to get it back into
shape?

I don't think it should be all that hard. At one point it worked pretty
well, so my guess is that the bit-rot that broke it isn't all that much
work to repair.

    Vern

19

I vaguely remember using it at one point a while ago, but maybe I didn't
use the broken parts?

I suspect you used it a number of years ago, pre-brokenness. Because right
now, it essentially doesn't work at all, I believe.

    Vern

20

Nobody said they are using it, some are using the pcap wrapper. So,
let's remove the native Endace/DAG support.

Robin