I'm performing testing for memory leaks and am running into an
interesting perftools error message:
Thread finding failed with -1 errno=1
Could not find thread stacks. Will likely report false leak positives.
Have memory regions w/o callers: might report false leaks
Leak check net_run detected leaks of 1520 bytes in 10 objects
The 1 largest leaks:
Leak of 1520 bytes in 10 objects allocated from:
If the preceding stack traces are not enough to find the leaks, try
running THIS shell command:
I can't tell what caused this error. Has anyone seen this before? This
is my first time running a memory leak test and this is happening for
all trace files, so I'm not sure if this is normal (guessing no).
When I use -m -M options with bro the output suggestions running “pprof” but the cli args don’t work:
pprof /opt/bro/bin/bro “/tmp/bro.17276.net_run-end.heap” --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv
pprof: invalid option – ‘-’
pprof: invalid option – ‘-’
pprof: invalid option – ‘-’
pprof: invalid option – ‘h’
pprof: invalid option – ‘h’
pprof: invalid option – ‘k’
pprof: invalid option – ‘-’
pprof: invalid option – ‘g’
pprof: invalid option – ‘-’
pprof: invalid option – ‘-’
pprof: invalid option – ‘g’
usage: pprof [-c|-b|-m|-t|-e|-i|-v] [-r] [-s] [-n num] [-f filename] [-p] [-l] [-d] [node numbers]
-a : Show all location information available
-c : Sort according to number of Calls
-b : Sort according to number of suBroutines called by a function
-m : Sort according to Milliseconds (exclusive time total)
-t : Sort according to Total milliseconds (inclusive time total) (default)
-e : Sort according to Exclusive time per call (msec/call)
-i : Sort according to Inclusive time per call (total msec/call)
-v : Sort according to Standard Deviation (excl usec)
-r : Reverse sorting order
-s : print only Summary profile information
-n : print only first number of functions
-f filename : specify full path and Filename without node ids
-p : suPpress conversion to hh:mm:ss:mmm format
-l : List all functions and exit
-d : Dump output format (for tau_reduce) [node numbers] : prints only info about all contexts/threads of given node numbers
root@bro-x64-01:~#
I'm performing testing for memory leaks and am running into an
interesting perftools error message:
Thread finding failed with -1 errno=1
Could not find thread stacks. Will likely report false leak positives.
Have memory regions w/o callers: might report false leaks
Leak check net_run detected leaks of 1520 bytes in 10 objects
The 1 largest leaks:
Leak of 1520 bytes in 10 objects allocated from:
If the preceding stack traces are not enough to find the leaks, try
running THIS shell command:
I can't tell what caused this error. Has anyone seen this before?
Running the `pprof` command that it gives (I usually omit —gv because I don’t have/want ghostview output) will help determine the cause. The `top` command in pprof usually gives me enough info to spot problems.
is my first time running a memory leak test and this is happening for
all trace files, so I'm not sure if this is normal (guessing no).
No, not normal, but if you happened to be testing git/master there was a recent leak introduced; now fixed:
Instead of “pprof”, which is sometimes associated w/ a package called TAU, check if you have something named "google-pprof”, which will be the actual tool associated w/ gperftools.
Thanks for the reply. This particular Bro package was the stable 2.3.2
release with an additional TCP analyzer I am testing, but perftools
reports leaks with the stable release of 2.3.2 on its own. Some
additional data for you if you're interested:
Using a 1.7GB pcap file ...
perftools reported Bro 2.3.2 as having a leak of 1216 bytes in 8 objects
perftools reported Bro 2.3.2 plus my analyzer as having a leak of 1368
bytes in 9 objects
perftools reported git/master as having a leak of 5136 bytes in 78 objects
Running the top command in pprof for all runs resulted in 0.0MB shown.
All runs did not indicate where the objects were allocated from.
I also ran all packages (standard 2.3.2, 2.3.2 with my analyzer, and
git/master) through valgrind and that found zero leaks in all runs. Do
you have an opinion on which tool (perftools or valgrind) is more
"accurate" with regard to reporting leaks? I'm not sure why perftools
is not finding thread stacks (unless they simply don't exist and the
reported leaks are false positives) ...
Regarding accuracy, I haven’t noticed much difference between valgrind and gperftools. Sometimes I prefer valgrind just to also check for other memory errors (e.g. invalid read/write, using uninitialized). AddressSanitizer is also a nice tool.
However, unless you’re using a suppression file, valgrind should be reporting a lot of “leaks” in Bro’s initialization routines that aren’t really concerning. With gperftools, Bro has been instrumented to only check for leaks in the main I/O loop, so leaks it reports are usually always concerning.
That's odd, I am using the configuration referenced on the Finding
Memory Leaks page: ./configure --enable-debug --enable-perftools
--enable-perftools-debug
I tried your configuration as well and receive the same results
(gperftools reports memory leaks but can't find thread stacks,
valgrind finds no memory leaks whatsoever). There must be something
wrong with one of my installations.
An empty debug.log is fine. It only has contents if at least one of the various debug streams is enabled via a -B<stream> flag when running bro. DebugLogger::streams in src/DebugLogger.cc has a list of stream names.
Thanks Jon. I have a simple program that definitely leaks memory and
valgrind reports this accurately, but I still cannot get it to work
with Bro. If you have time, would you copy / paste the command line
arguments you use to test Bro with valgrind?
Same for me, but at the least it should be reporting leaks in the init routines. Since I’m working on an analyzer I’m pretty determined to get this working accurately …
git clone --recursive git://git.bro.org/bro bro-tmp
cd bro-tmp
./configure --enable-debug --disable-perftools
cd build/
make -j8
. bro-path-dev.sh
valgrind --leak-check=yes ./src/bro -r ../testing/btest/Traces/http/get.trace
==12109== Memcheck, a memory error detector
==12109== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==12109== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==12109== Command: ./src/bro -r ../testing/btest/Traces/http/get.trace
==12109==
==12109==
==12109== HEAP SUMMARY:
==12109== in use at exit: 30,275,602 bytes in 434,261 blocks
==12109== total heap usage: 1,340,380 allocs, 906,119 frees, 115,312,199 bytes allocated
==12109==
==12109== 1 bytes in 1 blocks are definitely lost in loss record 2 of 5,455
==12109== at 0x4A06A2E: malloc (vg_replace_malloc.c:270)
==12109== by 0x3FB6080E91: strdup (in /lib64/libc-2.12.so)
==12109== by 0x6D3585: main (main.cc:533)
…
Wow, thanks Jon-- that helped a lot. When you test with perftools, do
you do anything different aside from change the configure options
(--enable-debug --enable-perftools-debug) and run without valgrind?
WARNING: Perftools heap leak checker is active -- Performance may suffer
Have memory regions w/o callers: might report false leaks
Leak check net_run detected leaks of 23 bytes in 23 objects
The 1 largest leaks:
Leak of 23 bytes in 23 objects allocated from:
@ 785d25
@ 6d61d2
@ 3fb601ed5d
@ 6b2359
If the preceding stack traces are not enough to find the leaks, try running THIS shell command:
If you are still puzzled about why the leaks are there, try rerunning this program with HEAP_CHECK_TEST_POINTER_ALIGNMENT=1 and/or with HEAP_CHECK_MAX_POINTER_OFFSET=-1
If the leak report occurs in a small fraction of runs, try running with TCMALLOC_MAX_FREE_QUEUE_SIZE of few hundred MB or with TCMALLOC_RECLAIM_MEMORY=false, it might help find leaks more repeatably
Memory leaks - aborting.
Aborted (core dumped)
Thanks ... I ran the same commands (also edited the Net.cc file) and
received this output:
WARNING: Perftools heap leak checker is active -- Performance may suffer
Thread finding failed with -1 errno=1
Could not find thread stacks. Will likely report false leak positives.
Have memory regions w/o callers: might report false leaks
Leak check net_run detected leaks of 475 bytes in 22 objects
The 1 largest leaks:
Leak of 475 bytes in 22 objects allocated from:
If the preceding stack traces are not enough to find the leaks, try
running THIS shell command:
Disregard my previous message ... I re-installed gperftools and it
seems to be working now. Thanks a lot for your help, you've saved me
(and hopefully others) lots of time and effort.