Issue when adding a field to files.log

Hi All,

I have written a script that extracts filetypes of interest, submits the extracted file to the cuckoo sandbox, and records the cuckoo task_id. I currently store this information successfully in the notice log, but would like to add an optional field to the files.log to store this task_id.

I have confirmed that I can add and populate the new files.log field with static values, but if I attempt to do this when calling an external program to handle the cuckoo submission (ie I use the ‘when’ block below), the value is not output in the log. The print statement within the when block, and notice.log output confirms the value is being populated, it is just not being written to files.log.

Any suggestions on what I might be doing incorrectly?

I have provided what I think are the relevant code extracts below, but am happy to provide more detail if that will assist:

Add the new field to the files.log

redef record Files::Info += {
cuckoo_id: int &optional &log;
};

Function that returns the cuckoo task_id

function submit_cuckoo(f: fa_file): int
{
local command = Exec::Command($cmd=fmt("%s extract_files/%s",tool,f$info$extracted));
return when ( local result = Exec::run(command)){
local id: int = to_int(result$stdout[0]);
return id;
}
}

Populate the new field

event file_state_remove( f: fa_file )
{
if (f$info?$extracted) {
when ( local id = submit_cuckoo(f) ){
f$info$cuckoo_id = id;
print fmt(“Cuckoo ID value set: %d”, f$info$cuckoo_id);
NOTICE([$note=File::Cuckoo_Submission,
$msg=fmt(“https://cuckoo/analysis/%s”, f$info$cuckoo_id),
$f=f]);
}
}
}

files.log extract

#fields ts fuid tx_hosts rx_hosts conn_uids source depth analyzers mime_type filename duration local_orig is_orig seen_bytes total_bytes missing_bytes overflow_bytes timedout parent_fuid md5 sha1 sha256 extracted cuckoo_id
#types time string set[addr] set[addr] set[string] string count set[string] string string interval bool bool count count countcount bool string string string string string int
1441526348.202595 FtBY2c3CsMMNsBdAil 192.168.1.xxx 192.168.1.yyy CKkqBYszNpSR6Bgaf HTTP 0 EXTRACT application/msword - 0.108599 -F 616960 616960 0 0 F - - - - HTTP-FtBY2c3CsMMNsBdAil.doc -

notice.log extract

#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc proto note msg sub src dst pn peer_descr actions suppress_for dropped remote_location.country_code remote_location.region remote_location.city remote_location.latitude remote_location.longitude
#types time string addr port addr port string string string enum enum string string addr addr port count string set[enum] interval bool string string string double double
1441526362.215942 CKkqBYszNpSR6Bgaf 192.168.1.yyy 33805 192.168.1.xxx 80 FtBY2c3CsMMNsBdAil application/msword http://192.168.1.xxx/files/test.doc tcp File::Cuckoo_Submission https://cuckoo/analysis/80 - 192.168.1.yyy 192.168.1.xxx 80 - bro Notice::ACTION_LOG 3600.000000 F- - - - -

What is most likely happening is that by the time your
external program returns its result, the log record has
already been written (without the cuckoo_id value) to files.log.

Thanks Daniel,

What you suggest makes sense and explains the behaviour I observed. I guess this leads me to the next thought - is there a way to delay the file getting written out, or an alternate File event that could be used to achive the outcome I am looking for?

I am happy pulling the data form the notice logs I am generating, but it seemed tidy to have this information in the file.log too if possible.

regards,

Boreham

My suggestion is to generate a whole new log with the cuckoo_id value
(cuckoo.log ?). The main advantage to doing it this way is that new
log entries will be written whenever Cuckoo analysis finishes-- you
won't need to delay files.log or continue to put cuckoo_id values in
notice.log. Additionally, if each entry in the new log has a UID, then
that's a very Brogrammatic way to correlate the cuckoo_id value to
entries in files.log.

Josh

This sounds like a good idea. The "Logging Framework" document in
the Bro Manual shows an example of how to create a new log stream
(look at the first part of the "Streams" section):
https://www.bro.org/sphinx/frameworks/logging.html

Hi Josh,

Yes - this would seem to be a sensible way to go. I’ll look in to the examples in the logging framework.

Best regards,
Boreham