Bif tuning / overhaul

Hi,

I've revamped the Bro bif language a bit. Mainly to add support for
namespaces and features that would be handy for my RPC/NFS analyzers.

I've also documented what one can do with bifs. Find it at:
  http://bro.icir.org/devel/bif-doc/
an overview of all of it:
  http://bro.icir.org/devel/bif-doc/example.html

The main changes (from current master version):

* support for policy-layer namespaces. Either using fully qualified
  names (MODULE::foobar) or using the "module XYZ;" statement.

* C/C++ variables, and functions have their own namespaces:
  BifConst for consts
  BifTypePtr for type declartions (RecordType*, EnumType*)
  BifEnum for C-enums derived from BiFs
  BifFunc for bif functions
  BifEvent for (some parts) of events. I haven't moved all of
               the event related C++ variables into this namespace,
               because this would substantial refactoring of existing
               code
  Question: Could also use BroConst, etc. instead of BifConst

* const:
  + can now use any type for consts (previously: only bools)
  + can now only declare but not define consts. You must define
    the const in bro.init. The bif only creates the netvar glue code.
    This will help streamline automatically generated documentation and
    it was necessary for supporting types other than bool

* forward type delcaration: can you declare but not define Bro types
  (records, sets, etc.) in BiF to make them available to C++. The bif
  only generates the netvar glue. The types must be defined in bro.init

Comments?

cu
Gregor
  etc.)

I've also documented what one can do with bifs. Find it at:
  http://bro.icir.org/devel/bif-doc/

Thanks for putting this together! I need to read a bit more carefully,
and I guess play with it, but overall this looks good to me. Nice
stuff.

  BifConst for consts
  BifTypePtr for type declartions (RecordType*, EnumType*)
  BifEnum for C-enums derived from BiFs
  BifFunc for bif functions
  BifEvent for (some parts) of events. I haven't moved all of
               the event related C++ variables into this namespace,
               because this would substantial refactoring of existing
               code

  Question: Could also use BroConst, etc. instead of BifConst

I actually like the Bif prefix, it clearly indicates where the stuff
is coming from. But how about using "BifType" instead of "BifTypePtr".
Types are always passed around as pointers anyway, so I don't see a
problem with leaving that hint out, and it makes the name more
intuitive.

  + can now only declare but not define consts. You must define
    the const in bro.init. The bif only creates the netvar glue code.

Is the reason for this just the work it would require to implement it
fully, or something else? Seems in principle bifcl could just as well
generate everything, with all the definitions (for consts, types,
etc.) being right there in the bif as well. That would be ideal as it
would follow the rule of "having things only in one place". (I'm not
asking you to implement it; just curious).

Robin

I actually like the Bif prefix, it clearly indicates where the stuff
is coming from. But how about using "BifType" instead of "BifTypePtr".
Types are always passed around as pointers anyway, so I don't see a
problem with leaving that hint out, and it makes the name more
intuitive.

Can do.
(I guess that's a leftover from an earlier version where all the
namespaces were called Bro* and BroType obviously already exists :wink:

  + can now only declare but not define consts. You must define
    the const in bro.init. The bif only creates the netvar glue code.

Is the reason for this just the work it would require to implement it
fully, or something else? Seems in principle bifcl could just as well
generate everything, with all the definitions (for consts, types,
etc.) being right there in the bif as well. That would be ideal as it
would follow the rule of "having things only in one place". (I'm not
asking you to implement it; just curious).

I agree that it would be great to be able to do all this stuff in BiF.
The reason why I haven't implemented it is indeed, that it would require
a lot of work to do it. Bro's syntax for defining types, and consts is
very rich and implementing that would require to add a good chunk of the
Bro syntax to Bifcl. We would probably have to rewrite most of the Bif
parser.
If we want that, we might want to consider using the Bro-parser and
extending it to support generation of C/C++ (basically have two modes
for the Bro-parser: BiF and script).

This would be something to keep in mind when we move towards compiling
Bro scripts (into HILTI) and/or when we think of adding "plugins" so
that analyzers can be developed externally and then loaded as a plugin.
Then BiF's could be just one such plugin.

cu
Gregor

Some more thoughts in this regard:

* Modules and export: functions, events, and enum that are defined in a
  bif in a namespace also need to be encapsulated in an "export". This
  is not super nice, but I don't know a better alternative.

* For consts, types defined in bro.init (and possibly then declared in
  a bif for inclusion in C/C++) you also need exports. I.e.,

    const FOO::foobar; # Doesn't work
    export { const FOO::foobar; } # Works

  Don't know whether there's a better way to solve that.

* Using namespace for global config variables and types, will also mean
  that we need multiple "module" statements in bro.init....

* "export" can be nested (although it doesn't make sense to do so).
   However as soon as the innermost export section ends, the "export"
   mode is switched off. I'd like to change that so that it will switch
   off when the outermost export section ends. (I need exports to
   support namespaces and modules for BiF, so depending on how users
   *bif.bro files, there might be nested export that will then not work
   as expected). This change is minor