I had the heartbeat branch running for a few hours (very successfully detecting activity!) and noticed it eventually had the manager worker consuming +70gb of memory. Wasn’t sure if the leak was from the heartbeat capability itself or something else along the current git repo… ymmv!
I may have experienced something similar; although I only have 64G of RAM on my worker node at present so my workers would reach about 7-8G committed memory within 5-15 minutes before being OOM-killed. I’ve had memory issues in the past on 2.2-release, so I can’t rule out my environment; although I’ve been problem free for the last month after moving to 2.2-184, so I just went back to that. When it was up and running I got a lot of good confirmed detections though.
Regards
Sorry about that. We’ve been battling a bug in git master for a while now that causes high memory use on the manager of clusters (production-only memory leaks are really difficult).
.Seth
Seth,
Any luck on a fix for this memory leak? Currently, this is causing my Bro server to come to a slow death.
Thanks,
John
John,
Try merging in this branch: https://github.com/bro/bro/tree/topic/jsiwek/ascii-log-memleak-fix
A number of people have tested it and reported that it fixed the memory issue.
--Vlad
It appears that the master branch was merged into this heartbeat
branch. Does this by chance include the memleak-fix merge you
mentioned? Is this possibly a test before merging these changes into
master its self?
Also, it has been a while since I did my install, and I can't recall.
If I do this on my master, then run the broctrl install, does it push
the new install to all the nodes? I know the configurations get
pushed out, but I can't recall if the entire install is pushed, or
just configuration files.
Thank you!
It appears that the master branch was merged into this heartbeat
branch. Does this by chance include the memleak-fix merge you
mentioned? Is this possibly a test before merging these changes into
master its self?
It does include the memory leak fixes that were mentioned, if you update
the branch to the current state these are included.
Also, it has been a while since I did my install, and I can't recall.
If I do this on my master, then run the broctrl install, does it push
the new install to all the nodes? I know the configurations get
pushed out, but I can't recall if the entire install is pushed, or
just configuration files.
The entire installation is pushed out.
Thank you!
You are welcome,
Johanna
I have pulled the latest branch, installed and pushed to my hosts. I
loaded the heartbleed as indicated, then I am testing with the
following site (https://filippo.io/Heartbleed/) so I can try and cause
a notice. After running the attack, I can't seem to get a notice
log.
So I figure either the attach generated by this site doesn't trigger
the script to insert a log, or I have something not configured right
still. Is there some way I can check to see that I am in fact on this
branch on all my nodes? Is there a specific version number or
something I can verify?
I can see the file in place, and the load statement in my local.bro,
so not really sure what else to check. Any assistance would be much
appreciated.
Alright, I have checked with multiple other websites, and it seems
that the first one I tried isn't detected. I have seen the notice
from a few other scans. I'll watch the memory and see if the fix that
was merged keeps it in check.
Hi,
that site uses the encrypted variant of the attack (hence it sends the
exploit heartbeat frames after encryption has begun).
In this case, it is more difficult to detect the attack than in the simple
case - we cannot just flag all heartbeats because that would introduce
a lot of false positive.
Thus, in case the attack is encrypted, you will only get notices if it was
successful (we still can determine that by comparing sizes), but not if
it was just attempted, sorry. There really is no good way around that.
So - you probably tested against a non-vulnerable server. If you test
against a vulnerable machine, you should get a notice in your log.
I think the heartbeat check by www.ssllabs.com always triggers - they
don’t start encryption before sending the heartbeats.
Johanna