tcmalloc large alloc

Hello,

We have been seeing some crash reports on some of our nodes, regarding a tcmalloc error. I was wondering if anyone else has seen this before and if anyone has any suggestions on what the cause might be. We are running Zeek 2.6. Here is an example stderr.log output from one of these crashes:

==== stderr.log

Myricom: Local timesource

listening on p2p2

tcmalloc: large alloc 1329594368 bytes == 0xc701c000 @ 0x7f72a12604ef 0x7f72a1280d56 0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5

tcmalloc: large alloc 1661992960 bytes == 0x11641c000 @ 0x7f72a12604ef 0x7f72a1280dad 0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5

/usr/local/bro/share/broctl/scripts/run-bro: line 110: 138751 Killed nohup “$mybro” “$@”

Thanks!

Hi,

I experienced the same, I think it might be related to corrupted
temporary files created by workers. An idea I had is that the corrupted
files are read with some wrong value and an allocation depend on it. The
following command solved the problem for me:

cleanup --all
deploy

Regards,

Les données à caractère personnel recueillies et traitées dans le cadre de cet échange, le sont à seule fin d’exécution d’une relation professionnelle et s’opèrent dans cette seule finalité et pour la durée nécessaire à cette relation. Si vous souhaitez faire usage de vos droits de consultation, de rectification et de suppression de vos données, veuillez contacter contact.rgpd@sgdsn.gouv.fr. Si vous avez reçu ce message par erreur, nous vous remercions d’en informer l’expéditeur et de détruire le message. The personal data collected and processed during this exchange aims solely at completing a business relationship and is limited to the necessary duration of that relationship. If you wish to use your rights of consultation, rectification and deletion of your data, please contact: contact.rgpd@sgdsn.gouv.fr. If you have received this message in error, we thank you for informing the sender and destroying the message.

We've seen evidence before that there is a file analyzer freaking out with particular files and attempting to do these very large allocations. Unfortuantely we still don't have concrete indications about exactly what is causing the problem. It would be helpful for us if you converted those offsets into symbolic procedure names. You can do it this way (just specify the correct location for your binary)...

addr2line -e /usr/local/bro/bin/bro 0x7f72a12604ef 0x7f72a1280d56 0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5

   .Seth

I work with Zach, here are the symbols:

$ addr2line -e /usr/local/bro/bin/bro 0x7f72a12604ef 0x7f72a1280d56 0x9623cf 0x9623ff 0x8d8c90 0x8d1b79 0x928352 0x92895f 0x928a71 0x9242bd 0x7b5908 0x7ff59f 0x7b535d 0x7b555f 0x7b3a98 0x8c422e 0x8c3a70 0x95d49e 0x95dc16 0x8c33cc 0x8c36f9 0x8c323f 0x8c18be 0x8bef32 0x95d352 0x5c61dd 0x676f75 0x677f1c 0x648a0f 0x914669 0x648ec5
??:0
??:0
/home/zeek/bro-2.6.1/aux/binpac/lib/binpac_buffer.cc:119
/home/zeek/bro-2.6.1/aux/binpac/lib/binpac_buffer.cc:529
/home/zeek/bro-2.6.1/build/src/file_analysis/analyzer/pe/pe_pac.cc:1900
/home/zeek/bro-2.6.1/src/file_analysis/analyzer/pe/PE.cc:26
/home/zeek/bro-2.6.1/src/file_analysis/File.cc:440
/home/zeek/bro-2.6.1/src/file_analysis/File.cc:481
/home/zeek/bro-2.6.1/src/file_analysis/File.cc:540
/home/zeek/bro-2.6.1/src/file_analysis/Manager.cc:167
/usr/include/c++/4.8.2/bits/basic_string.h:583 (discriminator 3)
/home/zeek/bro-2.6.1/src/analyzer/protocol/mime/MIME.cc:1230
/home/zeek/bro-2.6.1/src/analyzer/protocol/http/HTTP.cc:217
/home/zeek/bro-2.6.1/src/analyzer/protocol/http/HTTP.cc:161
/home/zeek/bro-2.6.1/src/analyzer/protocol/http/HTTP.cc:947
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/ContentLine.cc:174
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/ContentLine.cc:110
/home/zeek/bro-2.6.1/src/analyzer/Analyzer.cc:245
/home/zeek/bro-2.6.1/src/analyzer/Analyzer.cc:331
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:621
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:375
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Reassembler.cc:460
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP_Endpoint.cc:210
/home/zeek/bro-2.6.1/src/analyzer/protocol/tcp/TCP.cc:989
/home/zeek/bro-2.6.1/src/analyzer/Analyzer.cc:222
/home/zeek/bro-2.6.1/src/Conn.cc:271
/home/zeek/bro-2.6.1/src/Sessions.cc:769
/home/zeek/bro-2.6.1/src/IP.h:382
/home/zeek/bro-2.6.1/src/Net.cc:272
/home/zeek/bro-2.6.1/src/iosource/PktSrc.cc:263
/home/zeek/bro-2.6.1/src/Net.cc:315

The first two showing ??:0 makes sense b/c those are memory addresses. It looks like the PE analyzer might be the culprit but I'm not sure.
Thanks for your help!

Zander Work | Security Analyst | Oregon Research & Teaching Security Operations Center (ORTSOC)

A008 Kerr Admin Bldg | Corvallis, OR 97331 | Phone: 541-737-9800

Yep, I knew the first two would look like that. It's ASLR being applied to glibc function (which is fine and not what I was interested in anyway). It did end up showing what I expected it to. I'll look around a little bit and see if anything makes sense.

Thanks!
   .Seth

Thanks Seth, much appreciated!

Hey Seth,

Did you have a chance to look into this?

If anyone else has any input that would be helpful as well!

All the best,

There’s an issue here: https://github.com/zeek/zeek/issues/245

I believe the problem was fixed with https://github.com/zeek/zeek/commit/78dcbcc71ac09d3dd8a213f658ee8e794bb1bcd9 or https://github.com/zeek/zeek/commit/6598fe991d26bd15e483fcd96ea72bb161143d4e but it has not been confirmed yet,

Thanks Justin! I will see if we can do some testing on our end – If so I will report back.