Hi,
I am trying to use Bro sumstats framework. Based on the examples, I came up with the script shown at the end of the email. In the script, I am counting the number of http requests for each method+uri combination.
As dictated by the framework, I am calling observe for each request. At the end, I expected the total sumstats equal to the number of requests in my pcap. However, this doesn’t seem to be the case. I am trying understand if I made a mistake in how I am using the framework of if something else is going on.
For example, I ran the script on try.bro.org website using the http.pcap available there. Per my analysis, there should be 197 requests in the pcap. However, when I dump each of my stat into a log file, I expected the hits column from the log to add up to 197. However, that’s not the case. Running the script against my own pcap is giving different numbers from what I would expect.
Any help understanding the issue is appreciated… Thanks
Dk.
PS: you can copy paste this script in to try.bro.org website and run it against the http.pcap.
@load base/utils/site
@load base/frameworks/sumstats
module HttpStats;
export {
redef enum Log::ID += { LOG };
type Info: record {
ts: time &log;
method: string &log;
uri: string &log;
hits: count &log;
};
global update_http_stats: function(method: string, uri: string);
}
global scount: count = 0;
event bro_init() &priority=5
{
print “Creating HttpStats log stream and HTTP sumstats”;
flush_all();
Create the stream.
Log::create_stream(HttpStats::LOG, [$columns=Info, $path=“http-stats”]);
local r1 = SumStats::Reducer($stream=“http-stats”, $apply=set(SumStats::SUM));
SumStats::create([$name=“http-stats”,
$epoch=5sec,
$reducers=set(r1),
$epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) =
{
local r = result[“http-stats”];
local host_uri_vec = split_string(key$str, /,/);
local method = host_uri_vec[0];
local uri = host_uri_vec[1];
#local hits = double_to_count(floor(r$sum));
local hits = double_to_count(floor(r$num));
prep the record
local log_rec: Info = [$ts=ts, $method=method, $uri=uri, $hits=hits];
Log::write(HttpStats::LOG, log_rec);
}
]);
}
event bro_done()
{
Reporter::info(fmt(“scount=%d”, scount));
}
function update_http_stats(method: string, uri: string)
{
local key = cat_sep(",", “-”, method, uri);
scount += 1;
count URI hits.
SumStats::observe(“http-stats”, SumStats::Key($str=key), SumStats::Observation($num=1));
}
event http_request(c: connection, method: string, original_URI: string, unescaped_URI: string, version: string)
{
update_http_stats(method, unescaped_URI);
}
. Currently trybro limits log output to 200 lines for each file. It shows the first 100 and the last 100. I had always intended on making that more obvious and allowing that '200' parameter to be changed, but forgot all about it. It was mostly done as a performance optimization - the log output can be quite large and the result would either take too long to transfer to the client or the browser would freeze trying to render a table with 20k rows. The good news is that it is already a parameter on the backend, it just needs to be exposed to the api.
Never hurts to practice bro script debugging though. Turns out the old script was deleting entries from a table while iterating over it, which is undefined behavior in bro (and in many other languages).