Dear all,
I’m brand new to bro (just found out about it and tried yesterday) and I’m very intrigued by its capabilities.
The documentation says we can write outputs into databases BUT, as I got to the logging framework documentation, it seems the only “non file based” writer driver available is for sqlite.
I’m really interested in using a server based SQL instance (like postgresql, mysql or mariadb) AND a NoSQL service (mongodb or couchdb).
Are there any other writer drivers available (even if they are not officially supported / are part of non committed contributions)?
If not, can someone give me some pointers on how to develop extra writer drivers?
Thank you,
Luis Silva
There is a prototype of a postgresql writer in one a branch. However, it
is seriously outdated and will not compile with current Bro versions -- it
was either based on 2.2 or 2.1. And even for old bro versions it was
barely functional.
The best way to learn how to write logging writers is probably to take
a look at the already existing ones - they are mostly decapsulated from
Bro and quite easy to write.
Johanna
Hi Luis,
I had a similar requirement a while back and took a different approach
to get my data into Postgres by importing the output from the default
text writer. This was largely to keep things as simple as possible on
the sensor side.
See https://github.com/j-o-h-n-g/Mortimer/blob/master/broimport.py
The code is quite dirty in places, but might give you some ideas for
possible bro<->postgres type mappings.
John
Thanks John, that is very useful, though I was trying to avoid using an external import script as that will introduce a delay between the time the events happen and the registers get added to the DB.
Also, based on my experience, it can be pretty expensive to run an import script that parses through the data and translates it into insert calls.
Are the text writer logs created in append mode? If so, I could potentially have an external process that listens for new lines and adds things “in near realtime”.
Out of curisotiy, why didn’t you create a custom writer instead? …simplicity?
Thank you!
Luis
At the time simplicity and I had multiple remote sensors with
restricted network connectivity. I would rsync, or physically
transfer, the completed logs back to a central postgres server for
import and analysis. Real time alerting wasn't that important.
Getting the data into Postgres did facilitate the writing of some
useful SQL queries to spot odd/malicious behaviour. If I was doing it
again I probably investigate using Postgres Foreign Data Wrappers
instead.
John
Cool! And since you talked about multiple remote sensors, let me ask another question that I’m really curious about…
Lets imagine that I have a low end machine capturing traffic and want to send the prefiltered events into a more beefy remote machine for analysis and event capturing. Can I do that?
Based on bro’s Input framework, I believe I can redirect an entire tcpdump into it BUT, I want some filtering to happen upfront, though the MAIN processing work should be executed somewhere else.
From what I understood (based on this architectural description), my low end computer in charge of the sniffing would run the “manager” code and my beefy machine(s) would run the workers. Is that how I would set things up?
And who writes the outputs, is it the workers OR do the workers pass the result back to the manager?
Thank you,
Luis
I realized my last question was off topic from the original topic so I’m going to create a new thread for it.
Thank you!
Luis