Bro crash with "bro: out of memory in new"

Zeek
1

I am setting up a Bro IDS running on Freebsd6.3 AMD64 64 bit dual quad-core
processors.

Previous builds using ./configure
This build using ./configure --enable-int64 --enable-shippedpcap

I am getting crashes with the title "bro: out of memory in new".
I am sending the debuger output for one of these crashes. Please advise any
further information needed.

<snip>
#0 0x000000080131860c in kill () from /lib/libc.so.6
#1 0x000000080131749d in abort () from /lib/libc.so.6
#2 0x0000000000437302 in out_of_memory () at SSLInterpreter.cc:31
#3 0x0000000800fee45d in operator new () from /usr/lib/libstdc++.so.5
#4 0x000000000040e78c in std::vector<unsigned char, std::allocator<unsigned

>::reserve (this=0x303d168, __n=18446744071662796800) at

new_allocator.h:81
#5 0x0000000000429574 in binpac::SunRPC::RPC_Opaque::Parse (this=0x5449ca8,
t_begin_of_data=0x801459a00 "", t_end_of_data=0x801459a14 "G\bí\231\021Æ{H",
t_byteorder=20022828) at rpc_pac.cc:538
#6 0x0000000000429e77 in binpac::SunRPC::RPC_OpaqueAuth::Parse
(this=0x5cd3eb8, t_begin_of_data=0x8014599fc "", t_end_of_data=0x801459a14
"G\bí\231\021Æ{H", t_byteorder=0) at rpc_pac.cc:611
#7 0x000000000042a103 in binpac::SunRPC::RPC_Call::Parse (this=0x5b9a0b8,
t_begin_of_data=0x8014599e4 "", t_end_of_data=0x801459a14 "G\bí\231\021Æ{H",
t_context=0x3d67838, t_byteorder=0) at rpc_pac.cc:188
#8 0x000000000042b073 in binpac::SunRPC::RPC_Message::Parse
(this=0x552e040, t_begin_of_data=0x8014599dc "\v+4t",
t_end_of_data=0x801459a14 "G\bí\231\021Æ{H", t_context=0x3d67838) at
rpc_pac.h:155
#9 0x000000000042b1f4 in binpac::SunRPC::RPC_Flow::NewData (this=0x2f2f120,
t_begin_of_data=0x8014599dc "\v+4t", t_end_of_data=0x801459a14
"G\bí\231\021Æ{H") at rpc_pac.cc:1009
#10 0x000000000051f69d in RPC_UDP_Analyzer_binpac::DeliverPacket
(this=0x1772508, len=56, data=0x8014599dc "\v+4t", orig=44, seq=-2137894624,
ip=0x7fffffffdf38, caplen=0) at RPC.cc:608
#11 0x0000000000450073 in Analyzer::ForwardPacket (this=0x548d050, len=56,
data=0x8014599dc "\v+4t", is_orig=8, seq=-1, ip=0x7fffffffe480, caplen=64)
at Analyzer.cc:363
#12 0x000000000057396d in UDP_Analyzer::DeliverPacket (this=0x548d050,
len=56, data=0x8014599dc "\v+4t", is_orig=true, seq=-1, ip=0x7fffffffe480,
caplen=64) at UDP.cc:179
#13 0x000000000045fdef in Connection::NextPacket (this=0x5ff39ec,
t=3.8733205149138704e-317, is_orig=6, ip=0x2b0de40, len=64,
caplen=-2137894624, data=@0x7fffffffe3f0, record_packet=@0x7fffffffe3f8,
record_content=@0x7fffffffe3fc, hdr=0x80131862c, pkt=0x10d39 <Address
0x10d39 out of bounds>, hdr_size=0) at Conn.cc:241
#14 0x0000000000543a73 in NetSessions::DoNextPacket (this=0x133f7a8,
t=1216071185.658416, hdr=0x133f498, ip_hdr=0x7fffffffe480, pkt=0x8014599b2
"", hdr_size=14) at Sessions.cc:603
#15 0x0000000000543fe4 in NetSessions::NextPacket (this=0x133f7a8,
t=1216071185.658416, hdr=0x133f498, pkt=0x8014599b2 "", hdr_size=14,
pkt_elem=0x0) at Sessions.cc:294
#16 0x000000000050565e in net_packet_dispatch (t=1216071185.658416,
hdr=0x133f498, pkt=0x8014599b2 "", hdr_size=14, src_ps=0x133f458,
pkt_elem=0x0) at Net.cc:402
#17 0x000000000051233d in PktSrc::Process (this=0x133f458) at PktSrc.cc:211
#18 0x0000000000505d7e in net_run () at Net.cc:492
#19 0x00000000004344b1 in main (argc=9307256, argv=0x0) at main.cc:1008
(gdb)
<snip>

Thanks
Greg

2

This looks similar to a bug Tom Kho flagged a few months ago. (His concerned
traces without full payloads - do you have that?) I've appended a patch
for it, extracted from a larger patch, so it might be incomplete. Let me
know if it fixes it.

Note, we're quite close to releasing Bro 1.4, which includes this.

    Vern

Index: src/UDP.cc

3

Your host ran out of memory :slight_smile:

There are a couple of ways that I know of to compensate for this happening regularly. You can run a Bro cluster which spreads a good deal of the memory load across all of the machines in the cluster or you can run the prof.bro script and watch for the output of the global variable sizes in the prof.log file and subsequently tune your analysis to not store so much state.

  .Seth

4

Your host ran out of memory :slight_smile:

There are a couple of ways that I know of to compensate for this
happening regularly ...

Actually, in this case it's different. The analyzer is reading a length
value and allocating a buffer of that size, but the length is garbage and
the analyzer is coming up with a huge value which it then tries to malloc.

    Vern