Custom log file

Zeek
1

Hi all,

I want to generate custom log files.
For example, the columns of one log file should be like the following.
ts,uid,fuid,geo_location,idresp_h,idresp_p,method,status_code,trans_depth,response_body_len,mime_type,host,uri,referrer,source,filename,md5,sha1,extracted,flash_version.

Those fields are found in files.log and http.log.
To get those values I need connection, fa_file and fa_metadata records.
More precisely,
I need connection for resp_h and resp_p through conn_id record (c$id$resp_h and c$id$resp_p).
I need connection for ts, uid, trans_depth, method, status_code, response_body_len, host, uri, referrer, _flash_version through HTTP::Info record.
I need connection for geo location through lookup_location(resp_h).
I need fa_metadata for mime_type, since I extract only particular mime types and also I build the filename that is going to be extracted.
I need fa_file for fuid, source, filename, md5, sha1, and extracted through Files::Info record.

connection and fa_file records are accessible in event file_over_new_connection.
However, at this phase md5, sha1, extracted, and response_body_len values are not present.

For this reason, I used HTTP::log_http and Files::log_files events.
I can get all values from that events except resp_h and resp_p.
I use file_over_new_connection for that since I also extract geo location from here.
But in this case, due to the nature for network traffic that is not synchronic, resp_h and geo location values in my custom log file is erroneous.
How do you do that?

I read related parts of the documentation and source codes of the bro and alse reviewed the Bro archive for last one year.
I keep the post short in order not to be wordy for now.
I can send my script also.

Any help is quite appriciated,
Thanks,

2

Oh? Those two fields are part of the `id` field in the HTTP::Info record.

3

You are right. They are available also from there.
But it is not the solution of the problem.
Using both HTTP::log and Files::log_files makes inconsistency.
Because events for all packets occurring concurrently.
While I got tcp stream x in HTTP::log_http, I could get tcp stream 2 in Files::log_files.
So, I left this approach and developed two scripts, but they have problems also, details are below.

The data fields are extracted from the event file_state_remove(f: fa_file) in the following script.
https://pastebin.mozilla.org/8958179
There is two problem in here.
1 request is missing in the output of the script, because it returns 302 redirection HTTP status code, I think.
Alos, response_body_len is not available in here, since it will be available in connection_state_remove (c: connection).

In the second script,
The data fields are extracted from the event connection_state_remove(c: connection) in the following script.
https://pastebin.mozilla.org/8958182
There are two problems in here.
5 requests are missing in the output of the script, because in one connection, there is multiple http responses, I think.
Secondly, it is not possible to extract 5 fields, because they are only accessible via the f$info record.

There is 11 requests in the sample pcap.
http://wikisend.com/download/488246/test.pcap

4

Hi Justin,

HTTP::log_http and Files::log_files based approach is working now.

https://pastebin.mozilla.org/8958232

But I came to that point with trial-and-error method.
Here is the success story.
I should build filename at the event file_over_new_connection .
I should update filename with the extension in the file_sniff and call extract, md5, and sha1 analyzers in here.

I do not know why I need to extract filename at the file_over_new_connection method, but not in file_sniff or something else.
This script may work just for that sample, I need some guidance.

Thanks,