Hi All,
Rather than resurrect an old thread that I had I wanted to start a new one.
Our cluster has been mostly stable thanks to some suggestions (thanks Justin!).
That said, we’re still getting out of hand memory consumption and eventual swap usage. The (2) workers are bare metal, fully populated with 24 x 16GB memory modules. Attached is the relevant parts of node.cfg [1] (note, we WERE pinning the CPU’s but had a terrible time getting things to start up with them that way, so now we’re not. Packet loss and CPU usage is well within the acceptable range) [2].
The memory usage isn’t what’s confusing so much as the fact that it just never comes back down.
Where should we look?
Are you building zeek against jemalloc?
This plugin I put together for zeekctl makes it easy to enable jemalloc profiling and really understand the memory usage:
https://github.com/JustinAzoff/zeek-jemalloc-profiling
Sometimes the memory just doesn’t go down because malloc doesn’t necessarily return freed memory to the OS. I think jemalloc will, but might not if swap is enabled on the host.
Negative, perhaps I should?
bro@bro-master-1:/opt/zeek/etc$ zeekctl jeprof.check
Warning: ZeekControl plugin uses legacy BroControl API. Use
‘import ZeekControl.plugin’ instead of ‘import BroControl.plugin’
Error: unknown command ‘jeprof.check’
Wouldn’t others experience the same issues if the software just didn’t behave as expected? Is it that we’re undersized (and still eventually get to the point where the memory is needed)? Should I remove swap (so that it just starts dropping things rather than consuming swap?)
Hi Joseph,
We had similar memory utilization when we first deployed Zeek, our usage graph was like yours. We ended up disabling the scan detection script in local.zeek and memory usage stabilized afterwards. If you are loading the scan script, might be worth disabling it and see if it makes a difference? Changing @load misc/scan to #@load misc/scan in //share/zeek/site/local.zeek stops the script from loading when Zeek starts.
Carl Pearson | IT Security Analyst | University of Idaho
(208) 885-0957 | 875 Perimeter Drive MS 3155 | Moscow, ID 83844
