HIGH %MEM on Ubuntu 20.04 on rpi 4b

Zeek
1

Anyone else experiencing high MEMORY utilization? Across 9 devices all seeing over 50% utilization regardless of workload/zeek config.

$ /opt/zeek/bin/./zeekctl ps.zeek zeek
        USER         PID    PPID %CPU %MEM    VSZ   RSS TT       S  STARTED     TIME COMMAND
>>> localhost
   (+) root       99798   99791 38.5 **53.6** 4642156 2082572 ?     S 10:24:09 00:17:39 zeek

$ /opt/zeek/bin/./zeekctl capstats zeek 30
Interface             kpps       mbps       (30s average)
----------------------------------------
localhost/eth0        2.9        9.5

$ /opt/zeek/bin/./zkg --version
zkg 2.13.0-26
2

Hello @net_solv

$ /opt/zeek/bin/./zkg --version
zkg 2.13.0-26

Could you instead provide /opt/zeek/bin/zeek --version before and after ?

If you start Zeek standalone on an interface that has no traffic, do you also see it using as much memory? Are you using af_packet?

$ ip link add dummy0 type dummy
$ ip link set up dummy0
$ /opt/zeek/bin/zeek -C -i dummy0

Then check top for the memory usage for that process.

There has recently been a report of increased memory usage depending on the number of file descriptors - could you check ulimit -n?

3

This implementation is on the primary ubuntu 20.04 disk partition and not via any ‘container’ (aka docker, k8, etc.)

$ /opt/zeek/bin/zeek --version
/opt/zeek/bin/zeek version 5.2.2

zeek on dummy0 pid shows
$ps -p 94283 -o %mem | tail -n1 | tr -d ‘[:space:]’
5.7

$ ulimit -n
1024

$ sudo cat /opt/zeek/logs/current/stderr.log
listening on wlan1

zeek on wlan1 pid shows
$ ps -p 3249 -o %mem | tail -n1 | tr -d ‘[:space:]’
54.0

$sudo apt update
Get:4 /repositories/security:/zeek/xUbuntu_20.04 - openSUSE Download InRelease [1563 B]

Information on wlan1 is with NO workstations/users connected via wlan1… Additionally this instance has been updated/upgraded several times since initial install 12 months ago via apt update…

4

Hello - thanks for the input.

/opt/zeek/bin/zeek version 5.2.2
zeek on dummy0 pid shows
$ps -p 94283 -o %mem | tail -n1 ...
 5.7
$ ulimit -n
1024

Thank you. This was mostly to exclude the know issue. If you could also show rss the next time that would help. The absolute numbers are more informative generally.

Are you in a position to downgrade to 5.2.1 or 5.2.0 (or 5.0.9) to verify you see much lower memory usage in a previous release?

Otherwise, for runtime memory profiling you could attempt to use zeek-jemalloc-profiling, but it will require you to rebuild Zeek with --enable-jemalloc or use LD_PRELOAD. More detailed documentation about this is work in progress. It might be easiest to ask on Slack #general if you need guidance.

Hope this helps and we can figure out the root cause of the increase.

5

I can downgrade, what would be the best way to go about downgrading from Ubuntu 20.04 via cli? Are there any ‘issues’ I should be aware of by by doing a down grade? I could go back one version at a time to isolate which version started the %MEM issue… Any assistance would be helpful, as these are production environments and we would test on lab devices first. Thanks

6

I can downgrade, what would be the best way to go about downgrading from Ubuntu 20.04 via cli?

Assuming you have installed via the OBS Binary Packages, removing the old packages and installing just the zeek-lts package should allow you to downgrade from 5.2.x to 5.0.x.

$ ...make a backup of files in /opt/zeek you'd like to ensure don't get lost...
$ apt-get remove zeek* libbroker*
$ apt-get install zeek-lts
$ /opt/zeek/bin/zeek --version
/opt/zeek/bin/zeek version 5.0.9

I could go back one version at a time to isolate which version started the %MEM issue… Any assistance would be helpful, as these are production environments and we would test on lab devices first.

If you can reproduce it in a lab that would be great. See above how to go to 5.0.x. For any versions in between it seems it would require source build.

Thanks.

7

**BEFORE DOWNGRADE with ZERO USERS/DATASTREAM
$ /opt/zeek/bin/zeek --version
/opt/zeek/bin/zeek version 5.2.2

$ /opt/zeek/bin/./zeekctl ps.zeek zeek
USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND

localhost
(+) root 98529 98523 128 50.6 4222424 1964692 ? R 11:46:57 00:00:39 zeek

**AFTER DOWNGRADE with ZERO USERS/DATASTREAM
$ /opt/zeek/bin/zeek --version
/opt/zeek/bin/zeek version 5.0.9

$ /opt/zeek/bin/./zeekctl ps.zeek zeek
USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND

localhost
(+) root 9089 9083 3.1 6.1 1250712 240184 ? S 12:02:37 00:00:02 zeek

Do you know if Zeek is working on the %mem issue for v5.2.2+? I will build/step up builds to try and determine where %mem goes south. Thanks for your feedback/help.

8

Hey @net_solv - it does seems something clearly changed in your environment between 5.0 and 5.2…

We’d certainly be interested in what you’re finding! Your best bet will be using jemalloc, if you’re not in the position to rebuild Zeek, you can try setting LD_PRELOAD in node.cfg.

The troubleshooting instructions are now in the docs page. Feedback welcome.

Thanks!

9

Thanks for all your feedback/assistance… IF I find anything during debug(s) for MEM leak or buffer overflow, I will be sure to post back.