Bro & malloc implementations

Zeek
1

I wanted to share my experiences with bro and various malloc implementations. These are all running various versions of bro on SL 6.2 - a recompile of RedHat Enterprise Linux from Fermi Lab - https://www.scientificlinux.org/

  1. We are running a legacy Bro 1.5 installation currently supporting our old Instrumented SSHd infrastructure - http://code.google.com/p/auditing-sshd/ - we found that the standard malloc uses more memory than either of tcmalloc or jemalloc. I build bro with --enable-perftools to test tcmalloc, with a smaller memory footprint. To use jemalloc I just set LIBS=-ljemalloc - at this point we are using jemalloc, as it seems even a bit more memory thrifty than tcmalloc

  2. As part of the Instrumented SSHd infrastructure, we also have a perl script (ssllogmux) that runs a select loop accepting connections from all the Instrumented SSHd clients - typically several thousand at once. This is also in the Instrumented SSHd distribution. Perl is compiled to not use its own malloc, but to use the system. However, using the system malloc, this program would freeze after a day or so of operation. By using LD_PRELOAD, we forced it to use alternate malloc libraries - under both tcmalloc and jemalloc, it runs reliably.

  3. On another system, we ran Bro 2.0 (now running 2.1), and bro was (un)reliably freezing after a day or two - running but capturing no data when compiled with --enable-perftools - when compiled with --disable-perftools, it has been rock solid. I don’t know enough yet about Cmake to have it use another malloc implementation easily - hopefully someone else knows how to do that - I want to test jemalloc

So thats our current state of play - I would be interested in other folks experience.

2

Hi Jim:

Re: #3, here's what I did:

First, I ran './configure'

After successfully running configure, I opened up build/CMakeCache.txt and
found:
CMAKE_EXE_LINKER_FLAGS:STRING=' '

I modified that line to read:
CMAKE_EXE_LINKER_FLAGS:STRING=-L/path/to/jemalloc/lib/dir -ljemalloc

Then I did make clean / make / make install.

To verify bro was linked against jemalloc, I ran otool -L /path/to/bro and
verified that jemalloc was in the list of libraries I found there.

Also, other stuff I found to be useful:

* Exporting VERBOSE=1 before executing make will tell Cmake to dump the
complete compilation string to the console. This can be used to manually
verify that -ljemalloc is included in the flags when the bro binary is
being linked.

* The above will link all executables output during the build process with
the linker flags specified in CMAKE_EXE_LINKER_FLAGS. Since some of the
utilities generated earlier in the build are used later in the process,
the runtime library path needs to include libjemalloc before make is run.

Hope that helps,
Gilbert Clark