If memory serves me right, there was an old scanning detection script
and now there is a new one in ./scripts/policy/misc/scan.bro
The old one was discouraged on large clusters, is the new one better?
TL;DR to my surprise I have like 60Gbit of traffic here (OK, spikes
;-), millions of connections, insane amount of logs and I'm wondering
if I could enable it. It's not like I want to sacrifice lots of
performance, though. At best it will tell me that people are scanning
us 24/7, something that's quite obvious, but it would be a nice thing
to correlate and trace the attacker what he's doing, what other
services was he looking for before he started hammering some innocent
HTTP site and so on.
What do you think?
The old one was discouraged on large clusters, is the new one better?
Yes, vastly.
TL;DR to my surprise I have like 60Gbit of traffic here (OK, spikes
;-), millions of connections, insane amount of logs and I'm wondering
if I could enable it.
It should work just fine. We spent several years figuring out how to do it and extended SumStats a lot with that aim, but I haven’t yet heard of a network where it doesn’t work (although I’m sure I will now!).
A bit more information about why it works...
It was built on top of SumStats and was even one of the driving motivations for SumStats which gives us cluster transparency. There are a couple of reasons that SumStats works in general, even in crazy cases like scan detection. It uses lazy synchronization to wait until the end of an epoch (i.e. some time interval) to collect data from all of the nodes seeing traffic (workers). This means that the cluster is not synchronizing everything all the time. It’s only synchronizing results and defined intervals which greatly reduces the communication overhead. Additionally, all of the measurements that it does are composable (we can merge results from many different systems) which enables us to cope with traffic being split across processes and even machines like all of you are doing on clusters but then bring that data back and creates the final composed result which can then be checked against thresholds or you can have your own code run on the result.
Give it a try. I think it’ll surprise you. 
.Seth
Are you looking for examples where SumStats doesn't work from the latest
pull of Bro or a specific version? It's been my experience that it
depends heavily on the amount of networks configured in local nets.
Cheers,
Harry
Ah! I knew it. I’d be curious to learn conditions when and why (if we can figure it out) SumStats isn’t working. SumStats also hasn’t seen massive changes in a while so particular versions shouldn’t make much of a difference.
I’m really curious to figure out why more local nets would cause the issue. I’m mentally running through all of the cases where sumstats is used to think of places where more data is collected for local hosts. I’m at a loss right now. 
.Seth