Is there a way to select which intel “files” are sent to particular workers? Perhaps using the aux-script parameter in nodes.cfg?
We are running Bro 2.6.4. We are using two remote workers to collect specific data on much smaller machines. On our main cluster we are trying to load a +2GB file using the intel framework. The main cluster seems to handle this file with no problem. However, it is causing the “specialized” remote workers to crash “out of memory”.
I realize I’m not necessarily using the cluster feature as intended, but it has, thus far, been extremely convenient. I would like to not have to create a separate cluster.
Maybe if you describe your use case we can find something out. I don’t think you have 2GB or actionable intel
There’s also the input framework and the config framework, that can load data. It’s way easier to use the config framework on the 3.x line, as you don’t have to do the cluster magic yourself.
They just give you data so the matching would be on you and your scripts.
We are extending the concept of a bro/zeek cluster. In addition to our traditional cluster that analyzes large bandwidth taps, we started evaluating using additional ‘workers’ as sensors on servers to collect targeted data. For example, on a web proxy we collect web traffic, on DNS server we collect DNS queries, etc… We utilize the ‘aux_scripts’ feature in nodes.cfg of broctl to define capture filters appropriate for each service, which reduces load required to run those “sensors”. This concept has allowed us centrally manage all workers and aggregate data from many sources to one main pipeline.
In addition, we are ingesting many “security feeds” from many sources. Currently the cumulative size of all intel data files exceeds 3GB. The “traditional” cluster has no problem loading that intel. However, these small “sensors” do. A capture filter of, for example:
redef capture_filters += {
[“dns”] = “port 53”
};
will never match any intel with types: Intel::FILE_NAME, Intel::FILE_HASH, INTEL::URL. Allowing a bit more fine-grained control of how workers operate would allow us to maintain the centralized collection and control and scale our concept out to other applications without exploding resource requirements.
