Version: 2.0-907 -- Bro manager memory exhaustion

Development
1

I've been running 2.0-905 for ~25-26 hours. The manager's memory usage has
slowly crept up to 13 GB.

One thing of note - I'm using the ElasticSearch log writer. I see 3
possible scenarios for this memleak:

1) There is indeed a leak in master, potentially only triggered by
specific traffic,
2) There is a leak in the ElasticSearch log writer,
3) My ElasticSearch server can't keep up with the load, and the manager is
receiving logs faster than it can send them to the writer, so they just
queue up.

Has anyone else tried the current code over an extended period on live
traffic? Also, if anyone has any ideas to try to figure out where this
leak is occurring, please let me know. I'm going to switch back to ASCII
logs for a bit, and see what that's like.

  --Vlad

(Taking to bro-dev).

That's not a good sign for the manager ... It's possible that we have
a memory leak in there.

I just reran our leak tests and they didn't report anything (which is
good, but doesn't completely rule out any leaks).

I did see this though from valgrind:

    Object at 0x94e3410 of 68 bytes from an IgnoreObject() has disappeared

Does anybody know what valgrind it trying to tell me with that? Is it
a problem?

Robin

2

I’ve came to the conclusion that Bro cannot currently scale up to 100 workers for a single manager, threaded or not.

For now I’ve reverted back to the 2.0-stable build and have segmented the cluster into five parts, one per server with 20 workers on each. That has been stable, minus the occasional process stuck at 100% CPU. I’m going to try switching back to the dev track again with this segmented approach and see how that works.

–TC