# troubleshooting bro memory usage? **URL:** \ **Category:** Zeek\ **Created:** [August 2, 2013, 6:33pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762 "2013-08-02T18:33:03Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Aaron\_Gee-Clough](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@Aaron\_Gee-Clough](https://community.zeek.org/u/Aaron_Gee-Clough)\ **Post date:** [August 2, 2013, 6:33pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/1 "2013-08-02T18:33:03Z")
Hello, I've just put in two sensors running bro (with security onion), and am having trouble with the bro processes progressively growing in RAM usage, until they crash or become unresponsive. For example, I have one bro worker process right now that's reached 2.8 GB in 2 hours while watching a \< 100MB link. None of the other processes (manager/proxy/other workers) are anywhere near that...it's just this one worker. Are there any config options I can enable to attempt to find the cause of the memory leak? Also, since I'm confident the link I'm watching is missing some traffic (the span it's on is slightly mis-configured at the moment), where can I configure protocol timeouts? Thanks. aaron ---
**Author:** ![Aaron\_Gee-Clough](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@Aaron\_Gee-Clough](https://community.zeek.org/u/Aaron_Gee-Clough)\ **Post date:** [August 9, 2013, 7:30pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/2 "2013-08-09T19:30:48Z")
Hello, I've just come across something that implies Bro is caching all DNS resolutions that go past it ([https://bro-tracker.atlassian.net/browse/BIT-964](https://bro-tracker.atlassian.net/browse/BIT-964)). The bro systems I recently put in are in front of our main internal DNS resolvers, so almost all of the traffic they see is DNS resolution requests/answers. If Bro is caching all DNS, that would go a long way to explaining why bro's memory usage is continually increasing for my two sensors. Is there a way to disable this caching? (or have I mis-understood what bro's doing with DNS?) Thanks. aaron ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [August 10, 2013, 3:19pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/3 "2013-08-10T15:19:40Z")
That's unrelated. It's referring to DNS lookup requests happening at script land. We ran into a case once where someone had written a script that did two reverse hostname lookups for every connection that was established (don't do this, it's \*really\* not a good idea). Although I should point out that their Bro cluster was running quite well even in the face of that, but I don't think their DNS resolver was very happy about it. 🙂 In general, monitoring in front of a DNS resolver should be just fine.   .Seth ---
**Author:** ![Aaron\_Gee-Clough](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@Aaron\_Gee-Clough](https://community.zeek.org/u/Aaron_Gee-Clough)\ **Post date:** [August 11, 2013, 12:39pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/4 "2013-08-11T12:39:03Z")
> > Is there a way to disable this caching? (or have I mis-understood what > > bro's doing with DNS?) > > That's unrelated. It's referring to DNS lookup requests happening at script land. We ran into a case once where someone had written a script that did two reverse hostname lookups for every connection that was established (don't do this, it's \*really\* not a good idea). Although I should point out that their Bro cluster was running quite well even in the face of that, but I don't think their DNS resolver was very happy about it. 🙂 Heh. I'll keep that in mind. > In general, monitoring in front of a DNS resolver should be just fine. Hmm...that leaves me with my original problem, then: I have two vanilla securityonion installs (no custom .bro scripts added, just the ones that came with securityonion), watching just traffic to two different DNS resolvers...right now one of the worker parent processes (according to "broctl top") on each securityonion box grows monotonically in RAM usage until it gets killed by Linux (and is then restarted by broctl's cron job). Any ideas on where I should start looking to identify what's causing the worker to grow in RAM like that? Thanks. aaron ---
**Author:** ![Vlad\_Grigorescu2](https://avatars.discourse-cdn.com/v4/letter/v/c4cdca/32.png) [@Vlad\_Grigorescu2](https://community.zeek.org/u/Vlad_Grigorescu2)\ **Post date:** [August 12, 2013, 1:02am UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/5 "2013-08-12T01:02:31Z")
> I have two vanilla > securityonion installs (no custom .bro scripts added, just the ones that > came with securityonion), watching just traffic to two different DNS > resolvers What traffic rate do you see? > right now one of the worker parent processes (according to > "broctl top") on each securityonion box grows monotonically in RAM usage > until it gets killed by Linux (and is then restarted by broctl's cron job). How much RAM is in the box?   --Vlad ---
**Author:** ![Aaron\_Gee-Clough](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@Aaron\_Gee-Clough](https://community.zeek.org/u/Aaron_Gee-Clough)\ **Post date:** [August 12, 2013, 12:55pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/6 "2013-08-12T12:55:17Z")
> > I have two vanilla > > securityonion installs (no custom .bro scripts added, just the ones that > > came with securityonion), watching just traffic to two different DNS > > resolvers > > What traffic rate do you see? 95th percentile over a week (according to MRTG): Box 1: 34.6 Mbps. Box 2: 28Mbps > > right now one of the worker parent processes (according to > > "broctl top") on each securityonion box grows monotonically in RAM usage > > until it gets killed by Linux (and is then restarted by broctl's cron job). > > How much RAM is in the box? 16 GB. Both have 6-core 2.2GHz CPUs, also. Thanks. aaron ---
**Author:** ![Aaron\_Gee-Clough](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@Aaron\_Gee-Clough](https://community.zeek.org/u/Aaron_Gee-Clough)\ **Post date:** [August 13, 2013, 2:27pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/7 "2013-08-13T14:27:05Z")
All, I think I know what's causing this on the surface, but I'm unsure of the deeper cause. When I commented out the SecurityOnion bro scripts, bro's memory usage was stable and reasonable. So the problem was clearly coming from securityonion's scripts. I then started adding the SecurityOnion rules back in one by one, adding a ton of Reporter::warn statements, and watching the reporter.log. What I noticed was the securityonion hostname.bro script never completed \*if\* the device's hostname had a dash in it ("location-onion", for example). When I changed the server's hostname to not have a dash, the hostname script completed without issue. I suspect this means that the "hostname" and "interface" variables from the securityonion scripts weren't being initialized properly while trying to start up with a dashed hostname, doing who-knows-what when bro was told to add those variables to every logged event. Given that, I have an easy fix in the short term, which is to rename the box running securityonion to not have a dash in its hostname. What I'm confused by is why this would happen in the first place. (So I'm not clear yet on what patch to suggest to the securityonion folks to prevent this from coming up again.) The securityonion hostname.bro file does the following:     module SecurityOnion;     @load base/frameworks/input     export {              ## Event to capture when the hostname is discovered.              global SecurityOnion::found\_hostname: event(hostname: string);              ## Hostname for this box.              global hostname = "";     type HostnameCmdLine: record { s: string; };     event SecurityOnion::hostname\_line(description:     Input::EventDescription, tpe: Input::Event, s: string)              {              hostname = s;              system(fmt("rm %s", description$source));              event SecurityOnion::found\_hostname(hostname);              }     event add\_hostname\_reader(name: string)              {              Input::add\_event([$source=name,                                $name=name,                                $reader=Input::READER\_RAW,                                $want\_record=F,                                $fields=HostnameCmdLine,                                $ev=SecurityOnion::hostname\_line]);              }     event bro\_init() &priority=5              {              local tmpfile = "/tmp/bro-hostname-" + unique\_id("");              system(fmt("hostname \> %s", tmpfile));              event add\_hostname\_reader(tmpfile);              } The SecurityOnion::hostname\_line event never fires if the hostname has a dash in it (for example, if the contents of the tmpfile are "location-onion"). I see the add\_hostname\_reader event fire, but not the hostname\_line event. Do you all have any idea why that would fail if there's a string with a dash in the file? Is bro thinking it's an expression rather than a string? Two strings? Thanks for all the help so far. This has been hard to nail down. aaron ---
**Author:** ![Doug\_Burks](https://avatars.discourse-cdn.com/v4/letter/d/ee7513/32.png) [@Doug\_Burks](https://community.zeek.org/u/Doug_Burks)\ **Post date:** [August 13, 2013, 2:53pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/8 "2013-08-13T14:53:38Z")
Hi Aaron, There are definitely some issues with the hostname and interface scripts. My demo at Bro Exchange last week failed due to the hostname script, even though I put precautions in place which had always worked in the past. My hostname did include a hyphen, but I recorded a video later with the same VM (and same hostname) and everything worked fine: [http://youtu.be/0a2WDyBsxzk?t=2m36s](http://youtu.be/0a2WDyBsxzk?t=2m36s) I'll also mention that all of my production servers have a hyphen in the hostname and they work fine. Another thing I noticed in testing a few weeks ago in a VM was that if the VM had only a single CPU core the scripts were much more likely to fail. Increasing to 2 or more CPU cores resulted in much higher levels of success. Perhaps resource contention on Bro startup? Seth, I know you're going to rewrite these scripts for Bro 2.2, but do you have any ideas for troubleshooting in the meantime? Thanks! Doug ---
**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\ **Post date:** [August 13, 2013, 3:15pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/9 "2013-08-13T15:15:57Z")
Can you send a sample of those message? How much is a ton? 🙂 There's a known memory leak in Bro when the script interpreter reports certain errors in script code. If this happens very often, it could explain what you're seeing (unfortunately the leak is hard to fix, but the messages usually indicate a problem in the corresponding script in the first place). Robin ---
**Author:** ![Siwek\_Jon](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)\ **Post date:** [August 13, 2013, 3:26pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/10 "2013-08-13T15:26:28Z")
The hyphen-in-hostname might be a red herring when at least part of the issue is there's a bit of a race condition in the script -- the system() call to invoke `hostname` and put the output in a temporary file happens in a different background process, subject to the OS scheduler. So if that process gets scheduled after the input reader has already tried and failed to open the temporary file, the input reader won't automatically recover from that. I put a revision to the script you showed at [1] that \*should\* be a way to perform the same function without a race condition (though at the moment I'm not confident that the internals of the raw input reader are race-free in all cases, I'm looking in to some things). Still, I don't really know if this was actually the cause of your memory issues. - Jon [1] [https://gist.github.com/jsiwek/6222106](https://gist.github.com/jsiwek/6222106) ---
**Author:** ![Aaron\_Gee-Clough](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@Aaron\_Gee-Clough](https://community.zeek.org/u/Aaron_Gee-Clough)\ **Post date:** [August 13, 2013, 3:48pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/11 "2013-08-13T15:48:50Z")
I \*added\* a ton of Reporter::warn messages. Before this, bro was issuing one interesting error (see below), but I was basically adding lines like "script \ started with variables \", "script \ finished", etc to the reporter.log. So, the log messages looked like:     0.000000 Reporter::WARNING making tempfile:     /tmp/bro-hostname-ndOXgWQ3v52     /opt/bro/share/bro/securityonion/./hostname.bro, line 40     0.000000 Reporter::WARNING wrote hostname to tempfile     /opt/bro/share/bro/securityonion/./hostname.bro, line 42     0.000000 Reporter::WARNING called event to add hostname     reader /opt/bro/share/bro/securityonion/./hostname.bro, line 44     0.000000 Reporter::WARNING hostname reader starting on file:     /tmp/bro-hostname-ndOXgWQ3v52     /opt/bro/share/bro/securityonion/./hostname.bro, line 28     1376401730.326379 Reporter::INFO processing suspended (empty)     1376401730.326379 Reporter::INFO processing continued (empty)     1376401730.370328 Reporter::INFO processing continued (empty) What got me going this way was an error earlier that was:     0.000000 Reporter::WARNING Template value remaining in BPFConf     filename: /etc/nsm/{{hostname}}-{{interface}}/bpf-bro.conf     /opt/bro/share/bro/securityonion/./bpfconf.bro, line 99 which said to me that either the "hostname" or "interface" variable hadn't been initialized in the bro setup. aaron ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [August 13, 2013, 4:14pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/12 "2013-08-13T16:14:23Z")
I just looked through the scripts and I really don't know why that would happen. If anything, my guess is that Jon's probably right and there is a race condition that is causing it to fail in unpredictable ways. I'll start updating the scripts in that repository for 2.2 soon which might help a little. If I just update those in the master branch, could that cause any problems for SO?   .Seth ---
**Author:** ![Doug\_Burks](https://avatars.discourse-cdn.com/v4/letter/d/ee7513/32.png) [@Doug\_Burks](https://community.zeek.org/u/Doug_Burks)\ **Post date:** [August 13, 2013, 5:38pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/13 "2013-08-13T17:38:13Z")
Hi Jon, Thanks for the revised script! I'll try it out this week and see if it's more consistent. Thanks, Doug ---
**Author:** ![Doug\_Burks](https://avatars.discourse-cdn.com/v4/letter/d/ee7513/32.png) [@Doug\_Burks](https://community.zeek.org/u/Doug_Burks)\ **Post date:** [August 13, 2013, 5:38pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/14 "2013-08-13T17:38:23Z")
Updating in the master branch shouldn't cause any problems for SO since I packaged a static copy of the files and we're not actively pulling anything from the master branch. Thanks, Doug ---
**Author:** ![Tritium\_Cat](https://avatars.discourse-cdn.com/v4/letter/t/e8c25b/32.png) [@Tritium\_Cat](https://community.zeek.org/u/Tritium_Cat)\ **Post date:** [August 14, 2013, 4:28pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/15 "2013-08-14T16:28:18Z")
I’ve had this problem for too long. Wish I knew too. Seems each time it’s brought up on a mailing list the discussion gets hijacked and turns into feature requests or debates on new concepts and looses sight of the original problem. Keep hammering away. Good luck. ---
**Author:** ![Tritium\_Cat](https://avatars.discourse-cdn.com/v4/letter/t/e8c25b/32.png) [@Tritium\_Cat](https://community.zeek.org/u/Tritium_Cat)\ **Post date:** [August 14, 2013, 6:08pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/16 "2013-08-14T18:08:22Z")
Here’s a suggestion that has helped me in the past, disable all scripts except the SSH and SSH brute force detection. Basically you’re using process of elimination to find what aspect of Bro is not performing well in your environment. Turn on features of Bro one by one until you find which one is the culprit. It’s tricky to debug Bro from site to site because of different traffic profiles. –TC ---
**Author:** ![Aaron\_Gee-Clough](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@Aaron\_Gee-Clough](https://community.zeek.org/u/Aaron_Gee-Clough)\ **Post date:** [August 14, 2013, 8:12pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/17 "2013-08-14T20:12:43Z")
Thanks. Of the two boxes I have, one got better when I changed the hostname (have no idea why that helped, but it's been stable across reboots and restarts since then...perhaps luch). The other one I'm still working on. aaron ---
**Author:** ![David\_Kovar](https://avatars.discourse-cdn.com/v4/letter/d/f17d59/32.png) [@David\_Kovar](https://community.zeek.org/u/David_Kovar)\ **Post date:** [August 14, 2013, 8:27pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/18 "2013-08-14T20:27:29Z")
Greetings, Are you running Bro as part of Security Onion? I saw a discussion about SO issues with hostnames containing hyphens. -David ---
**Author:** ![Doug\_Burks](https://avatars.discourse-cdn.com/v4/letter/d/ee7513/32.png) [@Doug\_Burks](https://community.zeek.org/u/Doug_Burks)\ **Post date:** [August 14, 2013, 8:42pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/19 "2013-08-14T20:42:43Z")
Hi David, I think the hyphenated hostname was circumstantial evidence as the hostname/interface scripts were inconsistent even with non-hyphenated hostnames. Jon provided a workaround earlier in the thread that appears to be more consistent so far. I've packaged the updated scripts and uploaded to our "test" repo. Here's the email I sent to our testers last night: [https://groups.google.com/d/topic/security-onion-testing/KR\_Q-e-SjPQ/discussion](https://groups.google.com/d/topic/security-onion-testing/KR_Q-e-SjPQ/discussion) Thanks, Doug ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:41pm UTC](https://community.zeek.org/t/troubleshooting-bro-memory-usage/2762/20 "2022-05-06T15:41:08Z")