worker dies out of memory


I've had this problem pop up twice in the past couple weeks, where I
restart bro & learn that one of the workers has crashed. The message
that comes back in the crash report is:

listening on dag0:32, capture length 8192 bytes

==== stderr.log
1434739692.912574 processing suspended
1434739692.912574 processing continued
tcmalloc: large alloc 18446744072965615616 bytes == (nil) @
0x7f265e1ab8f5 0x81550d 0x88b986 0x88ba76 0x8147c9 0x814b54 0x81462b
0x812ea6 0x810265 0x88bb3a 0x5691a1 0x607698 0x608cd5 0x5d5461 0x85bd57
0x5d573b 0x532915 0x7f265d1aa76d 0x53b3bd
out of memory in new.
1434743469.366970 fatal error: out of memory in new.

==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited

==== .cmdline
-i dag0:32 -U .status -p broctl -p broctl-live -p local -p worker-17
local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

This is running on Ubuntu 12.04, bro 2.4, linux kernel 3.2.0-70, with
pcap compiled to use the Endace dag card, and bro compiled to use google
perftools. The box has 256GB of RAM.

Has anyone else seen this?



I ran into the same problem (with the DAG card) a few months ago.
My issue turned out to be that the firmware on the card needed an update.
Is your firmware up to date?


huh. It is not, because I'm running a custom firmware from Endace (card
kept causing the system to kernel panic). Time to bug Endace again, I guess.

Thanks for the pointer.