Hi all,
Running SecurityOnion and trying to implement Criticial Stack with
Bro, server running 24GB RAM the system becomes unresponsive in 30
seconds. All memory and swap is utilized by then. Any documentation
that show sizing of Bro and Critical Stack?
If I remove criticalstack from local.bro, it's back to normal.
Thanks
Monah
Sounds like you might be subscribed to a lot of lists...how big is your /opt/critical-stack/frameworks/intel/master-public.bro.dat file? Mine is 9 megs.
James
How many CriticalStack feeds are you subscribing to and against how much bandwidth are you monitoring?
I’ve heard a rough recommendation that anything more than 100k indicators can be pretty rough. We run with 90k against an average 1G traffic without any problems (14 workers).
-Dop
I subscribed to bambenekconsulting.com-DGA-Domains and the
master-public.bro.dat is 132MB in size.
I went with the most popular feed, I am open to suggestions as to what
feed to subscribe. I am interested in CNC alerts and malicious sites.
We have a 150MB pipe to the internet and around 70 users in the office.
I am running 1 worker though.
Thanks
Mike is correct.
When you create a collection the status indicator will actually warn your if the collection has too many indicators.
Try sorting the feeds by “Most Subscribers” and cherry pick from there. You can also try searching for terms like C&C, Botnet, Malware, Malicious etc via the search box at the top of the feeds page.
As for why have a feed with 700K+ indicators? There are quite a few folks that use the feeds outside of tools like Bro that consume and use all available indicators.
HTH.
Hey Baki,
Using the “Metrics” tab you can analyze the size in “count” of indicators by collection over time.
You may want to limit your deployments to between 100-200k indicators depending on cluster size, traffic, traffic types, etc.
There are three bambenek feeds available:
– precomputed dga feed (900k + elements)
– C&C IPs (260+)
– C&C Domains (330+)
Try building a collection with fewer items on it and then issuing an update.
If you look under your “collections” tab the “status” column will give you some feedback about the size of your collection.
Please feel free to open a ticket with us directly if you have any further problems.
V/r,
Liam Randall
Monah,
I don’t think that your subscriptions to intel feeds are what is causing this issue. I wouldn’t expect intel feeds to expand to fill that much memory space unless it were a truly massive amount of intel. You can certainly try reducing your subscriptions—I’ve definitely been wrong before. However, you should also check out a few other things:
How are you measuring memory utilization? The output of the linux ‘free’ command can be confusing to new users.
How big of a link are you trying to monitor?
Have you loaded any custom scripts into your Bro instance? It can be easy to fill a large amount of memory with Bro scripts.
What else is running on your Security Onion instance? A few of the tools distributed in Security Onion can be quite memory hungry.
Best of luck,
Stephen