Hi,
I just started using bro for offline traffic analysis. i don’t know which timers to tune to make the analysis of traces go faster. On some of traces, the analysis never finishes and it is like bro is waiting for some timer to expire.
any help?
kevin
Hi,
I just started using bro for offline traffic analysis. i don’t know which timers to tune to make the analysis of traces go faster. On some of traces, the analysis never finishes and it is like bro is waiting for some timer to expire.
any help?
kevin
I've been working with someone else having a problem similar to you. What would help most is if you were able to distribute one of the problematic tracefiles (hopefully, the smallest possible problematic file) so we could take a look at what's going on.
Also, what version of Bro are you running?
Thanks
.Seth
I just started using bro for offline traffic analysis. i don't know
which timers to tune to make the analysis of traces go faster. On
some of traces, the analysis never finishes and it is like bro is
waiting for some timer to expire.I've been working with someone else having a problem similar to you.
What would help most is if you were able to distribute one of the
problematic tracefiles (hopefully, the smallest possible problematic
file) so we could take a look at what's going on.
From what I've seen, I don't think the problem is only applicable to offline
tracefiles - it appears to happen on live traffic as well. My best guess is
that it is having a hard time when it only sees a portion of the full
traffic due to a busy link, thus making state tracking more problematic.
-- steve
From what I've seen, I don't think the problem is only applicable to offline
tracefiles - it appears to happen on live traffic as well.
Sure, that would simply mean that whatever's triggering it is (unsurprisingly)
showing up in the live traffic.
My best guess is
that it is having a hard time when it only sees a portion of the full
traffic due to a busy link, thus making state tracking more problematic.
That won't hang it or even partiuclarly burn up CPU. (We run in a lot of
environments with busy links, so know this from experience.)
We could realy use a trace that reproduces the problem to track this down.
Very likely it's a bug in an analyzer that's entering an infinite loop.
An alternative way to track it is to attach a debugger when it appears to
be wedged and get a traceback to see what it's doing. This will only be
effective if Bro has been built with ./configure --enable-debug.
Vern
Sure, that would simply mean that whatever's triggering it is (unsurprisingly)
showing up in the live traffic.
Yep! Stopping a daemonized BRO shows the same general symptoms where the
process does not die in a reasonable amount of time.
My best guess is
that it is having a hard time when it only sees a portion of the full
traffic due to a busy link, thus making state tracking more problematic.That won't hang it or even partiuclarly burn up CPU. (We run in a lot of
environments with busy links, so know this from experience.)
What I've seen is not so much the CPU hanging (though it was at 98% both in
and out of wedge), but BRO ends up processing a lot of timers and events at
this stage. Mostly rellated to conn.bro, but also in my case weird.bro,
port-name.bro, hot.bro events were firing.
It's not so much the amount of data I'm referring to but the data that makes
it to BRO. Assuming high random packet drops on a saturated link, stateful
tracking is problematic and most everything looks unatural because you're
not necessarily seeing the full picture. At least in my case, I had to turn
off ALL weird logging because it basically didn't apply to me.
Things did complete on a tracefile eventually, but very slowly. That
implied to me that it wasn't an infinite loop. The process looked something
like this (pardon the layman's view):
- Read pcap and process somewhat normally from start to finish
- Reach the end of the pcap as evidenced by the tracefile output
- Enter wedge state where the results take a very long time to complete
presumably due to processing of events/sessions still in state.
Unfortunately I'm not in a position to be able to provide tracefiles on this
particular issue.
-- steve
Unfortunately I'm not in a position to be able to provide tracefiles on this
particular issue.
A pity, as we do have quite a bit of machinery for analyzing performance
problems like this. One you could try directly is including profiling.bro
(and/or pkt-prof.bro) in your analysis scripts.
Vern