I’m rather new to Bro and have been asked to investigate why we’re not seeing very good throughput from Bro. We have added a significant number of scripts.
Are there any good script profilers out there that would tell me that I’m spending a lot of time in some script? I’ve considered using vTune or some other profiler, but that only gives me second-order profiling information.
Thanks,
–Scott
We have added a significant number of scripts.
Could you clarify more of what you mean here? I've seen lots of extremely inefficient scripts (and written quite a few myself!) and it's very possible you're doing something that isn't generally a good idea for operational use.
Are there any good script profilers out there that would tell me that I'm spending a lot of time in some script? I've considered using vTune or some other profiler, but that only gives me second-order profiling information.
Gilbert Clark is doing some profiling work but I'm not sure if that's going to address your needs. I'll let him comment on his work if he's reading this thread.
.Seth
Hi:
Are there any good script profilers out there that would tell me that I'm spending a lot of time in some script? I've considered using vTune or some other profiler, but that only gives me second-order profiling information.
Gilbert Clark is doing some profiling work but I'm not sure if that's going to address your needs. I'll let him comment on his work if he's reading this thread.
Unfortunately, I don't think anything I have would be in a good place to release at the moment. If this isn't a deterrent, feel free to get in touch off-list and we can chat: this isn't because the work is any big secret, but instead just because I'm a little hesitant to discuss anything tentative in a public setting. In my experience, discussions of this nature have a funny way of turning into commitments, and I'd hate to be making commitments that I wouldn't be able to later follow through on 
A few other things that might be more practical in the interim, though:
* There is a small script that ships with bro: aux/bro-aux/devel-tools/cpu-bench-with-trace. This script will find all the scripts loaded in local.bro and init-default and run bro with each of these scripts loaded in sequence, producing time output for each one. Assuming there's a trace available, this can be a simple way to help identify (relatively speaking) which scripts are introducing the most overhead.
* There are a few configuration settings that can be used to enable per-packet(ish) profiling: check out pkt_profile in init-bare.bro (http://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#id-pkt_profile_freq).
* Bro provides an option (-t) that will dump pretty much everything it's doing in script land into a log file. This is *incredibly* verbose and *very* expensive but, with a little scripting, can yield some useful information about what bro is doing. The overhead is high enough that I really doubt this could be used as an accurate timing mechanism, though.
* There is an option to enable segment profiling (http://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#id-segment_profiling) in prof.log. Segment profiling will show time spent in particular areas of bro. With that said, I have run into a few traces where bro crashes when the segment profiler is run. This could be because I'm doing something wrong, though
* If systemtap / dtrace is available and you don't mind getting your hands dirty (read: feel comfortable building bro and possibly tweaking a few things here and there), we can work through finding a few sane probe points in the code that might help with this task in a practical way.
Hope something in there is useful.
-Gilbert