I maintain blacklists of botnet C&C servers, spam sources etc. These are usually distributed as text files. Every once in a while, i need to update these by re-downloading them or better yet, by using rsync. In other cases, the database is too large to be locally maintained e.g. DNSBL and i would rather make an online query.
I want this process to be completely automated. That is to say, i want to provide Bro with a list of URL’s from where these lists can be obtained at the time of invocation. In my Bro script, i want to handle reading these files and also ‘refresh’ the lists say every 24 hours. Occasionally, i want to be able to make online queries about the ‘sanity’ of certain IP addresses.
Can i do this using Bro Script? If not, how do i go about doing this?
I highly recommend checking out the Collective Intelligence Framework (http://code.google.com/p/collective-intelligence-framework/) as a way to manage your blacklists. Of particular importance is its ability to store and share to authorized parties your org’s own custom blacklists in a seamless way with other blacklists.
Sounds like there are two parts to this: (1)
downloading/organizing/maintaining the information, and (2) then
getting it into Bro. The former might be best done outside of Bro with
something like CIF (as Martin wrote) or even just some simple shell
scripts. For the latter, we're working on much better support than is
currenty available: the next version will have a new input framework
that can read and parse files dynamically at runtime (including
dynamic updatess) and map the content into tables or events. It also
supports querying other sources like a DB or doing external queries.
There's a working prototype in git if you want to give it a try:
branch topic/johanna/input-threads. Feel free to send questions and
feedback to the development list, we're still working on finalizing
the specifics.