Cluster Best Practices

Zeek
1

I’m looking for feedback (or pointers to existing write-ups) on “best practices” for Bro cluster deployments. I’m planning to deploy workers to multiple geographic datacenters and I looking to weigh the pros/cons of two scenarios:

  1. Global Manager for all workers
  • Should there also be a global proxy or are there benefits to having one in each datacenter?
  1. Local Manager (per datacenter) for workers in that specific datacenter
  • Proxy would be local as well

A global manager would obviously be easier to manage/maintain but my concerns are:

  • Amount of “long-haul” traffic being generated to push log events to the manager
  • If the manager crashes are the workers queuing events until they re-connect to the manager?

In a scenario of separate managers per datacenter:

  • Can proxies still “sync” with each other? (e.g. push intel to workers watching similar traffic in each datacenter)

Thanks!
-Dave

2

I’m looking for feedback (or pointers to existing write-ups) on “best practices” for Bro cluster deployments. I’m planning to deploy workers to multiple geographic datacenters and I looking to weigh the pros/cons of two scenarios:

1) Global Manager for all workers
    - Should there also be a global proxy or are there benefits to having one in each datacenter?

Right now you can only have a single manager and your proxies can't currently be connected with specific workers so it's likely that if you setup one large cluster across multiple data centers that you would end up with workers connecting to proxies in the other data center.

2) Local Manager (per datacenter) for workers in that specific datacenter
    - Proxy would be local as well

I think my previous answer answered this as well. :slight_smile:

A global manager would obviously be easier to manage/maintain but my concerns are:
- Amount of “long-haul” traffic being generated to push log events to the manager
- If the manager crashes are the workers queuing events until they re-connect to the manager?

Right now, workers don't queue events like this. Events are delivered immediately to everyone subscribed to them (so if a host is crashed and not connected, it's obviously not subscribed at that moment).

In a scenario of separate managers per datacenter:
- Can proxies still “sync” with each other? (e.g. push intel to workers watching similar traffic in each datacenter)

Intel data is not synchronized though the proxies. I chose to do manual synchronization through events for the intel framework.

Your questions bring up a larger goal we have of creating hierarchical clusters. This is something where I think the current broctl overhaul is the first step toward that, but there is a lot more work to do. Ultimately what I'd like to see is that no matter how large your cluster is or how geographically dispersed, you can run your entire infrastructure as one Bro cluster. Unfortunately we aren't there yet and it's going to be a while before we are.

For now, I would setup separate clusters in each data center.

  .Seth

3

Thanks for the feedback Seth. In the scenario of running separate clusters in each data center; is it possible to sync Intel between clusters? For example, inbound email is load balanced across multiple data centers, as well as outbound client internet traffic. My goal is to extract URLs from inbound emails and push them into the Intel framework for alerting when outbound traffic matches (e.g. user clicked a link in an email), would that require all data centers to be in a single cluster?

-Dave

4

Thanks for the feedback Seth. In the scenario of running separate clusters in each data center; is it possible to sync Intel between clusters?

Unfortunately not without some work.

For example, inbound email is load balanced across multiple data centers, as well as outbound client internet traffic. My goal is to extract URLs from inbound emails and push them into the Intel framework for alerting when outbound traffic matches (e.g. user clicked a link in an email), would that require all data centers to be in a single cluster?

This sort of stuff is one of the many reasons we've been talking about doing hierarchical Bro clusters because it should make it straight forward to actually maintain intelligence data like you'd like to.

  .Seth