Custom log file

Hi all,

I want to generate custom log files.
For example, the columns of one log file should be like the following.

Those fields are found in files.log and http.log.
To get those values I need connection, fa_file and fa_metadata records.
More precisely,
I need connection for resp_h and resp_p through conn_id record (c$id$resp_h and c$id$resp_p).
I need connection for ts, uid, trans_depth, method, status_code, response_body_len, host, uri, referrer, _flash_version through HTTP::Info record.
I need connection for geo location through lookup_location(resp_h).
I need fa_metadata for mime_type, since I extract only particular mime types and also I build the filename that is going to be extracted.
I need fa_file for fuid, source, filename, md5, sha1, and extracted through Files::Info record.

connection and fa_file records are accessible in event file_over_new_connection.
However, at this phase md5, sha1, extracted, and response_body_len values are not present.

For this reason, I used HTTP::log_http and Files::log_files events.
I can get all values from that events except resp_h and resp_p.
I use file_over_new_connection for that since I also extract geo location from here.
But in this case, due to the nature for network traffic that is not synchronic, resp_h and geo location values in my custom log file is erroneous.
How do you do that?

I read related parts of the documentation and source codes of the bro and alse reviewed the Bro archive for last one year.
I keep the post short in order not to be wordy for now.
I can send my script also.

Any help is quite appriciated,

Oh? Those two fields are part of the `id` field in the HTTP::Info record.

You are right. They are available also from there.
But it is not the solution of the problem.
Using both HTTP::log and Files::log_files makes inconsistency.
Because events for all packets occurring concurrently.
While I got tcp stream x in HTTP::log_http, I could get tcp stream 2 in Files::log_files.
So, I left this approach and developed two scripts, but they have problems also, details are below.

The data fields are extracted from the event file_state_remove(f: fa_file) in the following script.
There is two problem in here.
1 request is missing in the output of the script, because it returns 302 redirection HTTP status code, I think.
Alos, response_body_len is not available in here, since it will be available in connection_state_remove (c: connection).

In the second script,
The data fields are extracted from the event connection_state_remove(c: connection) in the following script.
There are two problems in here.
5 requests are missing in the output of the script, because in one connection, there is multiple http responses, I think.
Secondly, it is not possible to extract 5 fields, because they are only accessible via the f$info record.

There is 11 requests in the sample pcap.

Hi Justin,

HTTP::log_http and Files::log_files based approach is working now.

But I came to that point with trial-and-error method.
Here is the success story.
I should build filename at the event file_over_new_connection .
I should update filename with the extension in the file_sniff and call extract, md5, and sha1 analyzers in here.

I do not know why I need to extract filename at the file_over_new_connection method, but not in file_sniff or something else.
This script may work just for that sample, I need some guidance.