Increased memory usage by Zeek..

Zeek
1

Hi All,

Couple of months ago I upgraded the Zeek cluster from 2.5 to 2.6.1 (compiled with the jemalloc support).
I have started seeing increased memory usage by the workers.

I have two physical sensors, each running 18 Zeek worker processes LB by PF_RING.
Not loaded any custom scripts, just the basic scripts that are enabled by default in local.bro (also have misc/scan disabled).

I just did a top on one of the boxes and here’s the output (specially two Zeek processes -13632, 13611 using >10% memory which is ~11G)
Also, attaching a weekly available free memory graph for the system.

Tasks: 455 total, 9 running, 443 sleeping, 0 stopped, 3 zombie
%Cpu(s): 18.3 us, 1.7 sy, 0.0 ni, 79.5 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 98783960 total, 32963660 free, 64807572 used, 1012728 buff/cache
KiB Swap: 4194300 total, 3572200 free, 622100 used. 33221356 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13589 bro 20 0 3662052 3.4g 73340 R 90.4 3.6 1072:47 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-5 local.bro broctl base/frameworks/cluster broctl/auto
13533 bro 20 0 1847972 1.6g 73188 S 50.3 1.7 1098:05 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-6 local.bro broctl base/frameworks/cluster broctl/auto
13512 bro 20 0 1291260 1.1g 73052 S 49.7 1.1 1080:30 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-1 local.bro broctl base/frameworks/cluster broctl/auto
13628 bro 20 0 2347952 2.1g 73328 R 49.0 2.2 1109:31 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-12 local.bro broctl base/frameworks/cluster broctl/auto
13516 bro 20 0 973260 799176 72844 R 47.0 0.8 1036:29 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-3 local.bro broctl base/frameworks/cluster broctl/auto
13539 bro 20 0 6374956 6.0g 73456 S 46.0 6.3 1147:08 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-2 local.bro broctl base/frameworks/cluster broctl/auto
13591 bro 20 0 865952 726516 73020 S 44.7 0.7 1052:29 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-4 local.bro broctl base/frameworks/cluster broctl/auto
13632 bro 20 0 12.2g 12.0g 73584 R 43.7 12.8 1068:17 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-15 local.bro broctl base/frameworks/cluster broctl/auto
13540 bro 20 0 2146844 1.9g 73348 R 41.4 2.0 1149:38 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-7 local.bro broctl base/frameworks/cluster broctl/auto
13611 bro 20 0 17.0g 16.7g 73404 S 39.7 17.8 1172:14 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-9 local.bro broctl base/frameworks/cluster broctl/auto
13640 bro 20 0 2624300 2.1g 73328 S 39.7 2.3 1043:50 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-18 local.bro broctl base/frameworks/cluster broctl/auto
13586 bro 20 0 3347044 3.1g 73468 S 39.1 3.2 1042:24 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-10 local.bro broctl base/frameworks/cluster broctl/auto
13641 bro 20 0 2274788 2.0g 73424 R 39.1 2.2 1029:58 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-17 local.bro broctl base/frameworks/cluster broctl/auto
13614 bro 20 0 1954780 1.7g 73188 S 38.4 1.8 995:00.54 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-13 local.bro broctl base/frameworks/cluster broctl/auto
13627 bro 20 0 2756520 2.5g 73288 S 38.4 2.6 1035:18 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-14 local.bro broctl base/frameworks/cluster broctl/auto
13638 bro 20 0 1206548 853056 72328 R 37.4 0.9 952:10.00 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-16 local.bro broctl base/frameworks/cluster broctl/auto
13623 bro 20 0 8998324 2.1g 73284 S 37.1 2.2 1073:31 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-11 local.bro broctl base/frameworks/cluster broctl/auto
13575 bro 20 0 871396 706148 73128 R 36.4 0.7 1028:30 /usr/local/bro/2.6.1/bin/bro -i p3p1 -U .status -p broctl -p broctl-live -p local -p worker-2-8 local.bro broctl base/frameworks/cluster broctl/auto
13336 bro 20 0 266244 133920 33388 S 12.6 0.1 400:27.62 /usr/local/bro/2.6.1/bin/bro -U .status -p broctl -p broctl-live -p local -p proxy-2 local.bro broctl base/frameworks/cluster broctl/auto

Any suggestions?

Thanks!
Fatema

weekly-mem-use.PNG

2

Hi!

I’ve been doing a ton of work in this space and have some tooling I’ve been working on to help track down things like this. I’m planning to have things ready for my ZeekWeek presentation, but if you have some time I can share the work-in-progress stuff with you and go over how to use it (which will help with the documentation bits that still need to be written).

The good news is I wouldn’t be surprised if this issue is already fixed or drastically better in 3.0 or master.

3

Biggest changes from 2.5.x to 2.6.x that I can recall are (1)
switching remote communication to use the new Broker library and (2)
enabling SMB analysis by default.

Had you manually enabled SMB in your previous 2.5.x deployment? If
not, you could see if disabling it helps:

    redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_SMB };

That's my first guess because we've recently seen/suspected (but not
yet fixed) some state management issues in the SMB analysis scripts
that might explain high memory usage.

- Jon

4

Hmm, I will disable the SMB analyzer in local.bro and see if it helps… Thanks Jon! :slight_smile: