Extracted files don't rotate

Zeek
1

Hey all,

So I recently changed the way I run bro at a site. Originally this was run via command line, now I have the below:

[logger]
type=logger
host=localhost

[manager]
type=manager
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=ethx

[worker-2]
type=worker
host=localhost
interface=ethx

extract_files shows up in worker-2. Here's the extract-files script:

global ext_map: table[string] of string = {
         ["application/x-dosexec"] = "exe",
         ["application/zip"] = "zip",
         ["application/msword"] = "xls",
         ["application/vnd.openxmlformats-officedocument.wordprocessingml.document"] = "docx",
         ["application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"] = "xlsx",
         ["application/vnd.openxmlformats-officedocument.presentationml.presentation"] = "pptx"

};

event file_sniff(f: fa_file, meta: fa_metadata)
         {
         if ( f$source != "SMTP" )
                 return;

         if ( ! meta?$mime_type || meta$mime_type !in ext_map )
                 return;

         local ext = "";

         if ( meta?$mime_type )
                 ext = ext_map[meta$mime_type];

         local fname = fmt("%s-%s.%s", f$source, f$id, ext);
         Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]);
}

In looking, I see that the files are accumulating and not rotating out. Anything I can do to troubleshoot this? Thank you.

James

2

Any takers on the below? Thank you.

James

3

Ah yes.. Extracted files aren't managed by anything. If you want files archived in a specific way you need to extract them to the full path that you want or setup a cron job to move+compress them periodically.

If you had something specific in mind I could probably whip up an example script for you.

4

Thanks Justin that's helpful. So as I look at my old setup I see that I indeed had move and compress manually on a cron job, so I'll just do that for the extract_files dir. Maybe a feature request down the road would be (maybe in broctl.conf) to be able to add "pre" rotate and "post" rotate scripts. Just a thought.

James

5

This has been a bit of a sticking point for quite a while. Part of the issue is the diversity in how clusters are run and managed. It's hard to create one solution which works for everyone's deployment.

I've been hoping to spend some time rejiggering how file extraction happens a little bit this year but I'd be glad to see anyone else beat me to it. It's a deceptively sneaky issue.

.Seth

6

Appreciate the feedback all.

James