I am running my own bro policy script (bro 1.4, debian lenny), together with a set of signatures that should be matched. After a few hours of runtime bro always and repeatably crashes, with the following error message:
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Bit by bit I stripped parts from my script in order to find the critical
part, and I ended up with a script as trivial as:
The critical part seems to be the signature matching. When including my
signatures, the error occurs. When outcommenting the last redef line, it
works without crashing (at least for a much longer time until I
terminate it deliberately, I cannot be totally sure that it hadn't crashed later). Note, that I don't even handle the signature
matches anymore, still the error occurs. My signature file is approx
100Kb, contains more than 600 signatures, and all of them look like:
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
That sounds like Bro is running out of memory. What's the process'
size just before it crashes and how much memory does the machine
have?
matches anymore, still the error occurs. My signature file is approx
100Kb, contains more than 600 signatures, and all of them look like:
If it's indeed memory exhaustion, then it looks like either a memory
leak in the signature engine or a general problem of handling the
many regexps. Generally, the engine can consume quite a bit of
memory due to the DFAs it builds incrementally. How do your regexps
look like? Do they contain many unanchored subparts (e.g.,
"foo.*bar")?
matches anymore, still the error occurs. My signature file is approx
100Kb, contains more than 600 signatures, and all of them look like:
If it's indeed memory exhaustion, then it looks like either a memory
leak in the signature engine or a general problem of handling the
many regexps. Generally, the engine can consume quite a bit of
memory due to the DFAs it builds incrementally. How do your regexps
look like? Do they contain many unanchored subparts (e.g.,
"foo.*bar")?
Yes, '.*' is massively used. Actually, that is the only regexp feature that is used. The patterns generally look like: ".*byte_seq1.*byte_seq2.*byte_seq3.*"
Upon termination, Bro will tell you the amount of memory your global
variables have accumulated. If you have a state-keeping problem in one
of those variables, you'll spot the issue there.
or another idea: load heavy-analysis.bro for reducing *_timeout.
heavy-analysis does the opposite - raises timeouts and increases resource
consumption. (It won't affect memory consumption due to massive .*
regular expressions either way.)
I'm guessing that these are indeed the problem, assuming there's no
leak somewhere. Having lots of such patterns is essentially the
worst case for a DFA-based pattern matcher (recall that Bro
internally combines many of these into a *one* regexp, which will
let the number of states explode).
Three things you could try:
(a) there is a tuning option for the signature engine which tells
Bro how many regexps to combine internally into Big Ones. It's
called sig_max_group_size and the default is 50. It might help to
reduce this quite a bit (e.g., 10 or 20).
(b) you could split each signature into several, one for each
component of the regexp (byte_seq1, byte_seq2, ...), and then either
chain these signatures with requires_signature condititions, or
raise an event for each one individually and correlate the matches
on the script-level to find out when all have matched. Both
approahces have the disadvantage that they don't consider the order
in which the subpatterns appear.
(c) this one is kind of scary. There's a configure option
--expire-dfa-states which enables some internal code to limit the
size of the DFAs Bro builds (by expiring less frequently used states
and recalculating them later if necessary). Enabling this has quite
a performance impact on the matching process but even more worse is
the fact that this option has most likely not been used by anybody
for >5 years ... I'd almost bet it's broken in some way but you can
still give it a try ...
I have applied suggestion (a) and set sig_max_group_size = 10. That
greatly improved the situation. I have been running Bro with my
signature set for a week now without a problem concerning memory
(before, it used to crash after hours), and from the prof.log it looks
like there is still a lot of headroom.
Great to hear, thanks for the update. Do you see any significant
change in CPU usage after the change? I'm wondering whether it might
make send to change the default value.