### Author:  [@net\_solv](https://community.zeek.org/u/net_solv)
#### Post date: [May 23, 2023, 3:16pm UTC](https://community.zeek.org/t/high-memory-utilization-50-on-zeek-running-on-raspberry-pi-4b/7038/1 "2023-05-23T15:16:17Z")
Anyone else experiencing high MEMORY utilization? Across 9 devices all seeing over 50% utilization regardless of workload/zeek config.
```console
$ /opt/zeek/bin/./zeekctl ps.zeek zeek
USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND
>>> localhost
(+) root 99798 99791 38.5 **53.6** 4642156 2082572 ? S 10:24:09 00:17:39 zeek
$ /opt/zeek/bin/./zeekctl capstats zeek 30
Interface kpps mbps (30s average)
----------------------------------------
localhost/eth0 2.9 9.5
$ /opt/zeek/bin/./zkg --version
zkg 2.13.0-26
```
---
### Author:  [@awelzel](https://community.zeek.org/u/awelzel)
#### Post date: [May 24, 2023, 9:18am UTC](https://community.zeek.org/t/high-memory-utilization-50-on-zeek-running-on-raspberry-pi-4b/7038/2 "2023-05-24T09:18:53Z")
Hello @net_solv
> $ /opt/zeek/bin/./zkg --version
> zkg 2.13.0-26
Could you instead provide `/opt/zeek/bin/zeek --version` before and after ?
If you start Zeek standalone on an interface that has no traffic, do you also see it using as much memory? Are you using af\_packet?
```auto
$ ip link add dummy0 type dummy
$ ip link set up dummy0
$ /opt/zeek/bin/zeek -C -i dummy0
```
Then check `top` for the memory usage for that process.
There has recently been a report of increased memory usage depending on the number of file descriptors - could you check `ulimit -n`?
>
>
> When using a docker image of Zeek 5.0.8 using Rocky 9 as the container OS and Ro…cky 9 as the host OS, I see heavy memory usage with each one of the zeek processes. Its around 8GB per process.
>
> jemalloc profiling:
> \`{"func": "monitoring\_thread\_loop", "direct\_mb": 8192.0, "direct\_pct": 98.8, "direct\_pct\_cumm": 98.8, "callees\_mb": 8192.0, "callees\_pct": 98.8}\`
>
> I also saw this in 5.0.7
---
### Author:  [@net\_solv](https://community.zeek.org/u/net_solv)
#### Post date: [May 24, 2023, 2:21pm UTC](https://community.zeek.org/t/high-memory-utilization-50-on-zeek-running-on-raspberry-pi-4b/7038/3 "2023-05-24T14:21:58Z")
This implementation is on the primary ubuntu 20.04 disk partition and not via any ‘container’ (aka docker, k8, etc.)
$ /opt/zeek/bin/zeek --version
/opt/zeek/bin/zeek version 5.2.2
zeek on dummy0 pid shows
$ps -p 94283 -o %mem | tail -n1 | tr -d ‘[:space:]’
5.7
$ ulimit -n
1024
$ sudo cat /opt/zeek/logs/current/stderr.log
listening on wlan1
zeek on wlan1 pid shows
$ ps -p 3249 -o %mem | tail -n1 | tr -d ‘[:space:]’
54.0
$sudo apt update
Get:4 [/repositories/security:/zeek/xUbuntu\_20.04 - openSUSE Download](http://download.opensuse.org/repositories/security:/zeek/xUbuntu_20.04) InRelease [1563 B]
Information on wlan1 is with NO workstations/users connected via wlan1… Additionally this instance has been updated/upgraded several times since initial install 12 months ago via apt update…
---
### Author:  [@awelzel](https://community.zeek.org/u/awelzel)
#### Post date: [May 24, 2023, 6:11pm UTC](https://community.zeek.org/t/high-memory-utilization-50-on-zeek-running-on-raspberry-pi-4b/7038/4 "2023-05-24T18:11:11Z")
Hello - thanks for the input.
> ```auto
> /opt/zeek/bin/zeek version 5.2.2
> zeek on dummy0 pid shows
> $ps -p 94283 -o %mem | tail -n1 ...
> 5.7
> $ ulimit -n
> 1024
>
> ```
Thank you. This was mostly to exclude the know issue. If you could also show `rss` the next time that would help. The absolute numbers are more informative generally.
Are you in a position to downgrade to 5.2.1 or 5.2.0 (or 5.0.9) to verify you see much lower memory usage in a previous release?
Otherwise, for runtime memory profiling you could attempt to use [zeek-jemalloc-profiling](https://github.com/JustinAzoff/zeek-jemalloc-profiling), but it will require you to rebuild Zeek with `--enable-jemalloc` or use `LD_PRELOAD`. More detailed documentation about this is [work in progress](https://github.com/zeek/zeek-docs/blob/topic/awelzel/doc-updates-for-6/troubleshooting.rst#jemalloc-memory-profiling). It might be easiest to ask on [Slack #general](https://app.slack.com/client/TSXGCHZE1/CSZBXF6TH) if you need guidance.
Hope this helps and we can figure out the root cause of the increase.
---
### Author:  [@net\_solv](https://community.zeek.org/u/net_solv)
#### Post date: [May 25, 2023, 12:55am UTC](https://community.zeek.org/t/high-memory-utilization-50-on-zeek-running-on-raspberry-pi-4b/7038/5 "2023-05-25T00:55:15Z")
I can downgrade, what would be the best way to go about downgrading from Ubuntu 20.04 via cli? Are there any ‘issues’ I should be aware of by by doing a down grade? I could go back one version at a time to isolate which version started the %MEM issue… Any assistance would be helpful, as these are production environments and we would test on lab devices first. Thanks
---
### Author:  [@awelzel](https://community.zeek.org/u/awelzel)
#### Post date: [May 25, 2023, 8:33am UTC](https://community.zeek.org/t/high-memory-utilization-50-on-zeek-running-on-raspberry-pi-4b/7038/6 "2023-05-25T08:33:53Z")
> I can downgrade, what would be the best way to go about downgrading from Ubuntu 20.04 via cli?
Assuming you have installed via the OBS Binary Packages, removing the old packages and installing just the `zeek-lts` package should allow you to downgrade from 5.2.x to 5.0.x.
```auto
$ ...make a backup of files in /opt/zeek you'd like to ensure don't get lost...
$ apt-get remove zeek* libbroker*
$ apt-get install zeek-lts
$ /opt/zeek/bin/zeek --version
/opt/zeek/bin/zeek version 5.0.9
```
> I could go back one version at a time to isolate which version started the %MEM issue… Any assistance would be helpful, as these are production environments and we would test on lab devices first.
If you can reproduce it in a lab that would be great. See above how to go to 5.0.x. For any versions in between it seems it would require source build.
Thanks.
---
### Author:  [@net\_solv](https://community.zeek.org/u/net_solv)
#### Post date: [May 25, 2023, 4:06pm UTC](https://community.zeek.org/t/high-memory-utilization-50-on-zeek-running-on-raspberry-pi-4b/7038/7 "2023-05-25T16:06:40Z")
\*\ ***BEFORE DOWNGRADE with ZERO USERS/DATASTREAM**
$ /opt/zeek/bin/zeek --version
/opt/zeek/bin/zeek version 5.2.2
$ /opt/zeek/bin/./zeekctl ps.zeek zeek
USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND
> > > localhost
> > > (+) root 98529 98523 128 **50.6** 4222424 1964692 ? R 11:46:57 00:00:39 zeek
\*\ ***AFTER DOWNGRADE with ZERO USERS/DATASTREAM**
$ /opt/zeek/bin/zeek --version
/opt/zeek/bin/zeek version 5.0.9
$ /opt/zeek/bin/./zeekctl ps.zeek zeek
USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND
> > > localhost
> > > (+) root 9089 9083 3.1 **6.1** 1250712 240184 ? S 12:02:37 00:00:02 zeek
Do you know if Zeek is working on the %mem issue for v5.2.2+? I will build/step up builds to try and determine where %mem goes south. Thanks for your feedback/help.
---
### Author:  [@awelzel](https://community.zeek.org/u/awelzel)
#### Post date: [May 25, 2023, 5:09pm UTC](https://community.zeek.org/t/high-memory-utilization-50-on-zeek-running-on-raspberry-pi-4b/7038/8 "2023-05-25T17:09:28Z")
Hey @net_solv - it does seems something clearly changed in your environment between 5.0 and 5.2…
> [@net\_solv](#):
>
> Do you know if Zeek is working on the %mem issue for v5.2.2+? I will build/step up builds to try and determine where %mem goes south. Thanks for your feedback/help.
We’d certainly be interested in what you’re finding! Your best bet will be using jemalloc, if you’re not in the position to rebuild Zeek, you can try setting LD\_PRELOAD in node.cfg.
The [troubleshooting instructions](https://docs.zeek.org/en/master/troubleshooting.html#jemalloc-memory-profiling) are now in the docs page. Feedback welcome.
Thanks!
---
### Author:  [@net\_solv](https://community.zeek.org/u/net_solv)
#### Post date: [May 25, 2023, 6:35pm UTC](https://community.zeek.org/t/high-memory-utilization-50-on-zeek-running-on-raspberry-pi-4b/7038/9 "2023-05-25T18:35:55Z")
Thanks for all your feedback/assistance… IF I find anything during debug(s) for MEM leak or buffer overflow, I will be sure to post back.