worker dies out of memory

Zeek
1

All,

I've had this problem pop up twice in the past couple weeks, where I
restart bro & learn that one of the workers has crashed. The message
that comes back in the crash report is:

listening on dag0:32, capture length 8192 bytes

==== stderr.log
1434739692.912574 processing suspended
1434739692.912574 processing continued
tcmalloc: large alloc 18446744072965615616 bytes == (nil) @
0x7f265e1ab8f5 0x81550d 0x88b986 0x88ba76 0x8147c9 0x814b54 0x81462b
0x812ea6 0x810265 0x88bb3a 0x5691a1 0x607698 0x608cd5 0x5d5461 0x85bd57
0x5d573b 0x532915 0x7f265d1aa76d 0x53b3bd
out of memory in new.
1434743469.366970 fatal error: out of memory in new.

==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited

==== .cmdline
-i dag0:32 -U .status -p broctl -p broctl-live -p local -p worker-17
local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

This is running on Ubuntu 12.04, bro 2.4, linux kernel 3.2.0-70, with
pcap compiled to use the Endace dag card, and bro compiled to use google
perftools. The box has 256GB of RAM.

Has anyone else seen this?

Thanks.

aaron

2

I ran into the same problem (with the DAG card) a few months ago.
My issue turned out to be that the firmware on the card needed an update.
Is your firmware up to date?

Michel

3

huh. It is not, because I'm running a custom firmware from Endace (card
kept causing the system to kernel panic). Time to bug Endace again, I guess.

Thanks for the pointer.

aaron