Hello all,
This is my first message to the mailing list. I was hoping someone could help me understand something regarding the HTTP module's http_request event. Specifically, I was hoping I could get access to the additional information added to the connection parameter by the HTTP module, but when the event is fired, my handler is unable to reference the information as it doesn't appear that it's there. The information I'm referring to is the following (from the bro/share/bro/base/protocols/httpd/main.bro file):
# Add the http state tracking fields to the connection record.
redef record connection += {
http: Info &optional;
http_state: State &optional;
};
When I try to get to the http field's host field, I get a "field value missing [WebRequests::c$http$host]" error.
Any thoughts?
Hmm, yknow it's been a while since I messed around in Bro code, but I
*think* the reason might be 'cos the host field is not filled in at that
point in the processing. It looks like it's not til the header is being
processed that it gets a value, in the "event http_header" part of
http/main.bro. The IP addresses might have values, though.
Just out of curiosity, can you talk about what you are trying to
accomplish here? Are you modifying the main.bro script, or are you
adding to local.bro, or what?
I appreciate the reply. I'm really at this point just trying to get comfortable with Bro scripting. While I understand that Bro will already log HTTP data for me, the end goal is to be able to log very specific things that I want. My first exercise is to approximate the output of the "urlsnarf" tool (part of the dsniff tools).
I think you're right about the host part. I guess at the time of the http_request event, the DNS resolution has already occurred and thus at this "layer" I can just see the IP address of the host receiving the request, and the "HOST" part of the HTTP header hasn't been seen yet. Is an alternative way to implement this to maintain the hostname from a previous event? I'm kind of groping in the dark here...
Instead of using the http request event, you’ll need to use an event that has all of the fields you want to use in the script. A quick way to identify those events is to look at the protocol analyzer page for common protocol events: http://www.bro.org/sphinx/scripts/proto-analyzers.html#bro-http
In many cases there are fields initialized in the connection portion of an event that are not immediately obvious. One way to see what fields are initialized in the event is to print the data to standard out (“print c;”) when running it in a local instance of Bro. You’ll quickly see what is initialized and what is not.
If you simply want all http data, you could use the http log event-- that’s generated as http logs are being sent to the logging framework.
event HTTP::log_http(rec: HTTP::Info)
{
print rec;
}
You'd probably want to get your information at the time the http.log
file entry is fired off, I think, or tie your output to a different event.
Have you seen the git repositories for Bro scripts of various sorts? And
of course there's this section of the Bro doc:
https://www.bro.org/sphinx/scripting/index.html
A couple years ago I was into modifying the Bro installation I was in
charge of (using Security Onion, a great set of tools). I'm not doing
that anymore (a huge family relocation) but I'm trying to stay current,
with an NSM installed in a VM at home.
Info records can have fields which are optional or get set at a later stage in processing than where user defined code is handled. It’s good practice to check if a field is present before using its value. An example of a check follows:
if (c$http?$host)
{
print “host field is there”;
}
-AK
Great info! I appreciate everyone's input and have kind of figured some of this out in the last couple of hours. Below is my current script if you're interested (and feedback is welcome). As you can see, I use the http_all_headers event to get the data I need. I had tried to use the log_http event earlier today but it wouldn't fire for reasons I'm not sure of.
@load base/protocols/http
module WebRequests;
export
{
redef enum Log::ID += { LOG };
type Request: record
{
ts: string &log;
source: addr &log;
dest: addr &log;
dest_port: port &log;
method: string &log &optional;
host: string &log &optional;
uri: string &log &optional;
referrer: string &log &optional;
user_agent: string &log &optional;
content_length: count &log &optional;
basic_auth_user: string &log &optional;
};
}
event bro_init()
{
Log::create_stream(LOG, [$columns = Request]);
}
event http_all_headers(c: connection, is_orig: bool, hlist: mime_header_list)
{
if (is_orig)
{
local req: Request;
req$ts = strftime("%Y/%m/%d %H:%M:%S", c$http$ts);
req$source = c$id$orig_h;
req$dest = c$id$resp_h;
req$dest_port = c$id$resp_p;
if (c$http?$method) req$method = c$http$method;
if (c$http?$host) req$host = c$http$host;
if (c$http?$uri) req$uri = c$http$uri;
if (c$http?$referrer) req$referrer = c$http$referrer;
if (c$http?$user_agent) req$user_agent = c$http$user_agent;
if (c$http?$request_body_len) req$content_length = c$http$request_body_len;
if (c$http?$username) req$basic_auth_user = c$http$username;
Log::write(LOG, req);
}
}