local.bro causing memory leak

Zeek
1

Hey Ben,

So, if the whole purpose of doing file renaming was just for Splunk streaming,

then I wonder why it won’t work for you to just have forwarder keep monitoring the log files in the current Bro log dir.

We are also a Splunk shop, and index some of our Bro logs into Splunk using Splunk forwarder running on our Bro manager, and just monitoring logs/current/ dir for the types of logs we want to index.

It’s a very basic setup, works without any issues on our side. wonder why it would create a problem in your situation. Hmm (or I might have mis-interpreted the problem :slight_smile: ).

Fatema.

2

I’m wondering that now too.

At some point we realized that events weren’t making it all the way to splunk. But I couldn’t tell you why. I’m going to be taking a step back and re-evaluate why the forwarder didn’t work.

I also figured out that using path_func in this way is a very BAD IDEA. I’ve spent a couple days crawling through the source code for the logging framework and testing some things.

For every unique filename, a new thread will be started and a new writer will be created on a per file basis. The bad news is, there is no way to reap these threads when they are no longer needed.

The only “in framework” process that will close file descriptors and reap process threads is rotation. Even if you enable rotation, files can still slip through, because rotation seems to only be effective against the current writer.

I don’t know if this breaks a contract outlined in the docs, but it seems like if this is an intended use of path_func then this is a bug that should be fixed.

The only way that I could resolve the problem in bro alone, would be to author a custom log writer that would name the files in the way I wanted, and close these dangling file descriptors.

It’s a pretty complicated problem, and I’m planning to abstract it away buy using the features in a splunk forwarder, or using Apache NiFi to manage the logs if that fails.

Thanks,

Ben

3

I'm wondering that now too.

At some point we realized that events weren't making it all the way to splunk. But I couldn't tell you why. I'm going to be taking a step back and re-evaluate why the forwarder didn't work.

I also figured out that using path_func in this way is a very BAD IDEA. I've spent a couple days crawling through the source code for the logging framework and testing some things.

For every unique filename, a new thread will be started and a new writer will be created on a per file basis. The bad news is, there is no way to reap these threads when they are no longer needed.

The only "in framework" process that will close file descriptors and reap process threads is rotation. Even if you enable rotation, files can still slip through, because rotation seems to only be effective against the current writer.

Well they should all be current and rotation will work. path_func is not normally used for splitting a file based on time, it's used for doing things like

    Log::remove_default_filter(HTTP::LOG);
    Log::add_filter(HTTP::LOG, [
        $name = "http-directions",
        $path_func(id: Log::ID, path: string, rec: HTTP::Info) = {
            local l = Site::is_local_addr(rec$id$orig_h);
            local r = Site::is_local_addr(rec$id$resp_h);

            if(l && r)
                return "http_internal";
            if (l)
                return "http_outbound";
            else
                return "http_inbound";
        }
    ]);

in this case, internal,outbound, and inbound are all current writers and they will all get rotated.

I don't know if this breaks a contract outlined in the docs, but it seems like if this is an intended use of path_func then this is a bug that should be fixed.

It's not a crazy idea, you're just the first person to ever do that.

The only way that I could resolve the problem in bro alone, would be to author a custom log writer that would name the files in the way I wanted, and close these dangling file descriptors.

I can think of 2 solutions:

1) Just turn rotation back on and set the rotation interval to 5 minutes and disable compression. Point splunk at

    /usr/local/bro/logs/*/*.log

The end result will be the same, the only downside is all data in splunk will have a 5 minute lag.

2) Enable rotation, but override

Log::default_rotation_postprocessors

the default runs this:

redef Log::default_rotation_postprocessors += { [Log::WRITER_ASCII] = default_rotation_postprocessor_func };
...
# Default function to postprocess a rotated ASCII log file. It moves the rotated
# file to a new name that includes a timestamp with the opening time, and then
# runs the writer's default postprocessor command on it.
function default_rotation_postprocessor_func(info: Log::RotationInfo) : bool
    {
    # If the filename has a ".gz" extension, then keep it.
    local gz = info$fname[-3:] == ".gz" ? ".gz" : "";

    # Move file to name including both opening and closing time.
    local dst = fmt("%s.%s.log%s", info$path,
            strftime(Log::default_rotation_date_format, info$open), gz);

    system(fmt("/bin/mv %s %s", info$fname, dst));

    # Run default postprocessor.
    return Log::run_rotation_postprocessor_cmd(info, dst);
    }

if you redef Log::default_rotation_postprocessors to be empty, bro should close the old filenames and then "rotate" them.