All,
I think I know what's causing this on the surface, but I'm unsure of the deeper cause. When I commented out the SecurityOnion bro scripts, bro's memory usage was stable and reasonable. So the problem was clearly coming from securityonion's scripts. I then started adding the SecurityOnion rules back in one by one, adding a ton of Reporter::warn statements, and watching the reporter.log. What I noticed was the securityonion hostname.bro script never completed *if* the device's hostname had a dash in it ("location-onion", for example). When I changed the server's hostname to not have a dash, the hostname script completed without issue.
I suspect this means that the "hostname" and "interface" variables from the securityonion scripts weren't being initialized properly while trying to start up with a dashed hostname, doing who-knows-what when bro was told to add those variables to every logged event.
Given that, I have an easy fix in the short term, which is to rename the box running securityonion to not have a dash in its hostname. What I'm confused by is why this would happen in the first place. (So I'm not clear yet on what patch to suggest to the securityonion folks to prevent this from coming up again.)
The securityonion hostname.bro file does the following:
module SecurityOnion;
@load base/frameworks/input
export {
## Event to capture when the hostname is discovered.
global SecurityOnion::found_hostname: event(hostname: string);
## Hostname for this box.
global hostname = "";
type HostnameCmdLine: record { s: string; };
event SecurityOnion::hostname_line(description:
Input::EventDescription, tpe: Input::Event, s: string)
{
hostname = s;
system(fmt("rm %s", description$source));
event SecurityOnion::found_hostname(hostname);
}
event add_hostname_reader(name: string)
{
Input::add_event([$source=name,
$name=name,
$reader=Input::READER_RAW,
$want_record=F,
$fields=HostnameCmdLine,
$ev=SecurityOnion::hostname_line]);
}
event bro_init() &priority=5
{
local tmpfile = "/tmp/bro-hostname-" + unique_id("");
system(fmt("hostname > %s", tmpfile));
event add_hostname_reader(tmpfile);
}
The SecurityOnion::hostname_line event never fires if the hostname has a dash in it (for example, if the contents of the tmpfile are "location-onion"). I see the add_hostname_reader event fire, but not the hostname_line event. Do you all have any idea why that would fail if there's a string with a dash in the file? Is bro thinking it's an expression rather than a string? Two strings?
Thanks for all the help so far. This has been hard to nail down.
aaron
