working with Broccoli

Hello, everyone,

I started this question over on the security onion list, but it was pointed out that this list is probably the better venue for Bro questions. Anyway, here’s what I’m trying to do: I’m trying to pull all of the dns resolution events for several DNS servers off the wire (from several different locations, so there will be multiple listening boxes involved), so that I can put the data in a database for alerting/reporting/etc. That separate database (Arcsight) isn’t something I can replace/remove at present (that’s a political decsion that I can’t change), but I do want to get this DNS information into it as cleanly as possible.

My first assumption was that I would use Security Onion & Bro for this, since the dns.log file contains exactly what I’m looking for. I had considered simply dumping the dns.log file (or syslog sending it out), but broccoli came to my attention as a way to make this process easier (allowing me to have one central script to format the data for arcsight rather than having to teach arcsight how to read bro logs)…but I’ve had a lot of trouble getting the broccoli python bindings to do what I’m expecting of them.

Here’s where I am at the moment:

  1. I have added the following to the bro config:

@load policy/frameworks/communication/listen
@load base/protocols/dns/main

global api_test_log = open_log_file("/tmp/apitest.log");

redef Communication::nodes += {
[“api”] = [$host=127.0.0.1, $connect=F]
};

global dns_via_api: event(orig_h: addr, resp_h: addr, start_time: time, ans: dns_answer, reply: strin
g);

event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string) &priority=-4
{
event dns_via_api(c$id$orig_h, c$id$resp_h, c$start_time, ans, reply);
print api_test_log, fmt("%s dns answer received, creating custom event.", reply);
}

The idea here is to add a custom event that only has the specific DNS bits I’m interested in, and in theory my python script would subscribe to that custom event. The api_test_log file is only there for me to verify that the custom event is firing…once this is working properly I would remove that bit. This seems to be working (I see records in the test log) from the bro side, but I still haven’t got the python side doing what I want yet.

  1. I have a python script that does this:

import broccoli

dns_answer = broccoli.record_type(“answer_type”, “query”, “qtype”, “qclass”, “TTL”)

@broccoli.event(broccoli.addr, broccoli.addr, broccoli.time, dns_answer, broccoli.string)
def dns_via_api(orig_h, resp_h, start_time, ans, reply):
print reply

bc = broccoli.Connection(“127.0.0.1:47763”, connect=False)
bc.subscribe(“dns_via_api”, dns_via_api)
bc.connect()

while True:
bc.processInput()
time.sleep(1)

This gets something (which is progress), but the script crashes with:

File “/opt/bro/lib/broctl/broccoli.py”, line 87, in wrapped_f
new_args += [instantiate(btype, val, type)]
File “/opt/bro/lib/broctl/broccoli.py”, line 408, in instantiate
return _Factories[src_type](val, dst_type)
File “/opt/bro/lib/broctl/broccoli.py”, line 282, in _factory
return record(dst_type, vals)
File “/opt/bro/lib/broctl/broccoli.py”, line 297, in init
for (key, val) in zip(type.fields, vals):
AttributeError: ‘NoneType’ object has no attribute ‘fields’

Could any of you give me a pointer as to what I’ve missed here?

Thank you.

aaron

import broccoli

dns_answer = broccoli.record_type("answer_type", "query", "qtype", "qclass", "TTL")

@broccoli.event(broccoli.addr, broccoli.addr, broccoli.time, dns_answer, broccoli.string)
def dns_via_api(orig_h, resp_h, start_time, ans, reply):
print reply

bc = broccoli.Connection("127.0.0.1:47763", connect=False)
bc.subscribe("dns_via_api", dns_via_api)
bc.connect()

while True:
bc.processInput()
time.sleep(1)

Try removing the bc.subscribe() call. I think it's only for use with non-decorated event callbacks, but you've already got the @broccoli.event decoration there.

    Jon

Wed Mar 06 2013 05:31:59 PM EST from “Siwek, Jonathan Luke” jsiwek@illinois.edu Subject: Re: [Bro] working with Broccoli

import broccoli
`dns_answer = broccoli.record_type("answer_type", "query", "qtype", "qclass", "TTL")`
@broccoli.event(broccoli.addr, broccoliaddr, broccoli.time, dns_answer, broccoli.string)
def dns_via_api(orig_h, resp_h, start_time, ans, reply):
print reply
`bc = broccoli.Connection("127.0.0.1:47763", connect=False)` `bc.subscribe("dns_via_api", dns_via_api)` `bc.connect()`
while True:
bc.processInput()
time.sleep(1)
``

Try removing the bc.subscribe() call. I think it's only for use with non-decorated event callbacks, but you've already got the @broccoli.event decoration there.
``
Jon

That was it. the callbacks are working beautifully now. Thank you very much.

aaron

Someone’s going to have to explain what this is and add a field to the Info record.

#info$util = 100.0*((res$user_time + res$system_time) - (last_res$user_time + last_res$system_time))/(now-last_ts);

This is just an estimate of the cpu utilization to process the packets. If you seeing lost packets and this number is a 100% then you run out of the processing power for that worker.

Most of my dropped packets are due to running out of cpu. This mostly happens during denial of serves attach when some is try to use TACC resole to generate icmp replies. Were carefully not generate icmp messages in most cases.