Update on log management

Development
1

As per Gilbert's request, I'm moving this discussion over to the dev list.

Last night I sent out a small set of scripts to the list which I've
been using to help leverage Bro logs with other systems, namely by
sending them to our central syslog as well as logging the data to a DB
and putting a basic web query interface on the data.

Gilbert, you said that you've got a similar project for log snarfing
going, but I am a bit of a n00b on your git system and I can't find
your topic branch. Can you provide a link to it?

Can you guys please lay out what the basic goals, roadmap, and
timeline are for all Bro output? Please be clear about things you
would *like* to do versus things which will actually be production
ready in the next few months.

Thanks,

Martin

2

Right now we have the logging framework's writer plugin API ready to go. We don't have any concrete plans for when or what writer plugins we are going to write but personally I'm very interested in a PostgreSQL plugin and you've gotten me interested in getting a syslog writer created if TCP syslog does in fact allow for extremely long messages.

To be honest it pained me a little bit when I saw your code to insert the ascii logs into a database (however I can't fault anyone for doing things out of operational necessity). The benefits to outputting logs directly to a database from Bro is that you get to take advantage of logging framework features. You can filter out log lines that you don't want to go to certain outputs or remove certain fields. For instance, if we had a PostgreSQL writer plugin and you only wanted SSL sessions that didn't successfully validate put into the database, you could do this (keeping in mind that due to the lack of a postgresql writer, this code very much does not work)....

event bro_init()
  {
  Log::add_filter(SSL::SSL, [$name="failed_to_pg",
    $writer=POSTGRESQL_WRITER,
    $path="ssl_invalid_cert_session", # This equates to the table name in postgresql
    $pred(rec: SSL::Info) = { return (rec$validation_status != "ok"); }]);
  }

All of the SSL logs would still be written to the normal (ascii by default) logs but all of the sessions using invalid certificates would also be sent to postgresql.

In terms of future plans we're really just at the point where we need more writer plugins, most of the rest of the code is finished. Does that answer your questions?

  .Seth

3

To be honest it pained me a little bit when I saw your code to insert the ascii logs into a database (however I can't fault anyone for doing things out of operational necessity).

LOL! Yes, this is very much out of operational necessity. I would
love to hear how other incident response teams are using Bro in
production (beyond PoC setups), as right now, it doesn't integrate
with anything enterprise, and therefore becomes much more limited than
it should be given all of its sophisticated abilities.

The benefits to outputting logs directly to a database from Bro is that you get to take advantage of logging framework features.

Yes, we all want this to be integrated inside Bro. The reason that my
script snarfer works efficiently, ugly and inelegant as it is, is
because you are guaranteed that the effort and errors of dealing with
the output from Bro is not Bro's problem. This is critical for things
like database errors, network lag, and CPU overhead. Please, please
make sure that you make these proposed writers async and in their own
thread. Do not make the same mistakes Snort made by having the output
in the main thread. Output is a very easy task to delegate to a child
thread or fork, and because Bro does an incredible amount of logging,
you don't want to be wasting precious CPU cycles in the network
detection thread on output. My single Bro instance is overwhelmed on
any traffic over 200 Mb/sec on a decent machine.

event bro_init()
{
Log::add_filter(SSL::SSL, [$name="failed_to_pg",
$writer=POSTGRESQL_WRITER,
$path="ssl_invalid_cert_session", # This equates to the table name in postgresql
$pred(rec: SSL::Info) = { return (rec$validation_status != "ok"); }]);
}

All of the SSL logs would still be written to the normal (ascii by default) logs but all of the sessions using invalid certificates would also be sent to postgresql.

Yep, this is good stuff. I'd settle for this working on files, as I
can limp along on my scripts until the Pg plugin is ready.

In terms of future plans we're really just at the point where we need more writer plugins, most of the rest of the code is finished. Does that answer your questions?

Almost. The next question is: and then what? So let's say you've got
all of this data in Pg. Do you have anyone working on a frontend for
this? Or is accessing the data Bro puts out considered wholly
out-of-scope for the Bro project? Again, I'd love to hear from other
IR team leads as to how they are using this data to either trigger or
supplement investigations.

4

The benefits to outputting logs directly to a database from Bro is that you get to take advantage of logging framework features.

Yes, we all want this to be integrated inside Bro. The reason that my
script snarfer works efficiently, ugly and inelegant as it is, is
because you are guaranteed that the effort and errors of dealing with
the output from Bro is not Bro's problem.

The "normal" Bro deployment is as a cluster at this point anyway where the manager is dedicated to notice handling and logging (as you've noticed, standalone instances basically suck for anything over 80Mbps). Also, Gilbert has been spending the summer threading the logging framework and I *think* his branch is probably close to being integrated. We basically planned on threading the logging framework from the start for all of the reasons that you mentioned. :slight_smile:

I started inserting Bro logs into PostgreSQL a long time ago at OSU too with my bro-dblogger project...
  GitHub - sethhall/bro-dblogger: Utility for logging data from the Bro Intrusion Detection System directly to PostgreSQL <- Deprecated! This project is only here for historical curiosity now.

If you read the README for that project closely, you can see way back then that I was already heading down the path that we ended up going down with the new logging framework. :wink:

Yep, this is good stuff. I'd settle for this working on files, as I
can limp along on my scripts until the Pg plugin is ready.

The rest of the filter actually works already except for the $writer. :slight_smile:

In terms of future plans we're really just at the point where we need more writer plugins, most of the rest of the code is finished. Does that answer your questions?

Almost. The next question is: and then what? So let's say you've got
all of this data in Pg. Do you have anyone working on a frontend for
this? Or is accessing the data Bro puts out considered wholly
out-of-scope for the Bro project?

We've been poking around at various people and places trying to figure out what a Bro interface would look like and do. I suspect we aren't too far off from movement in this area, but we have no plans yet.

Again, I'd love to hear from other
IR team leads as to how they are using this data to either trigger or
supplement investigations.

I'm hoping to have a couple of people talk about this specifically at the Bro workshop.

  .Seth

5

Martin:

http://git.bro-ids.org/bro.git/tree/refs/heads/topic/gilbert/log-util:/aux/log-util

should take you to the python library when opened in a browser. Keep in mind, though, that this code won't be practically usable until we finish fixing up the ASCII header and I revise that code accordingly; I used a prototype header format to write / test this code.

See bro-logtool in that directory for a simple script I've been using to play with the library.

Also, re:

The "normal" Bro deployment is as a cluster at this point anyway where the manager is dedicated to notice handling and logging (as you've noticed, standalone instances basically suck for anything over 80Mbps). Also, Gilbert has been spending the summer threading the logging framework and I *think* his branch is probably close to being integrated. We basically planned on threading the logging framework from the start for all of the reasons that you mentioned. :slight_smile:

Working revision is in topic/gilbert/log-threads. There's some work left to do here, but the current code does seem to pass basic testing.

--Gilbert

6

The "normal" Bro deployment is as a cluster at this point anyway where the manager is dedicated to notice handling and logging (as you've noticed, standalone instances basically suck for anything over 80Mbps). Also, Gilbert has been spending the summer threading the logging framework and I *think* his branch is probably close to being integrated. We basically planned on threading the logging framework from the start for all of the reasons that you mentioned. :slight_smile:

Cool, got it.

The rest of the filter actually works already except for the $writer. :slight_smile:

Excellent!

We've been poking around at various people and places trying to figure out what a Bro interface would look like and do. I suspect we aren't too far off from movement in this area, but we have no plans yet.

Ok.

I'm hoping to have a couple of people talk about this specifically at the Bro workshop.

Yes, I'm still hoping management will let me attend. In the meantime,
if there are any incident responders on the list who wouldn't mind
sharing with me off-list in what capacity they use Bro, it would be
appreciated!