Combining HTTP scripts?

Here's another "should we change the default" question: the HTTP
scripts have always been inconsistent with others in the way they
split functionality across a set of scripts; and one then has to load
the right ones depending on which pieces are desired.

I think we should unify this so that in the future there's just a
single HTTP script to load, potentially then with options for
selecting specifics. Seth's http-ext already does much of that in
fact.

Is there a consensus that that's the right way to go?

Robin

I'd say that's probably the right way to go. I think some confusion has come from those scripts being separated as they are.

Based on how the logging framework's shaping up though, we may have some more options about how this is structured. I think you may be right though that the ultimate answer could be configuration options in the http script. It would match well for the auto generated documentation too.

  .Seth

Is there a consensus that that's the right way to go?

In general I like that. However, where do you draw the line? Processing
individual headers and/or capturing transferred entities gets expensive.
It's certainly reasonable that a default config gets response processing
along with request processing; but I'm not sure about what other stuff it
should include, due to load / log space considerations.

    Vern

That a good point and one that I've certainly debated back and forth with myself quite a bit. Here's what I came down to in my http-ext script...
  https://github.com/sethhall/bro_scripts/blob/master/logging.http-ext.bro#L24

Not all of those fields are filled in by default (ex. md5, which is the md5 sum of the response body) but with the combination of record extensions and the logging framework, it should be really easy for users to add their own data to this record while maintaining the separation between shipped scripts and locally written ones.

For cases like writing out the http response body, the normal file writing and printing functions and statements are still there so that the key-value logging framework can be bypassed.

  .Seth