We run 2 redundant Zeek clusters, which provides some safeguards against individual host or process failure, and virtually eliminates downtime due to periodic maintenance.
However, ingesting terabytes of logs daily into Splunk eats into our Splunk license. Has anyone tried deduplicating redundant Zeek logs before sending them to Splunk (or Elastic Stack, Humio, Graylog, etc.)?
My other thought is to simply not send one redundant set to Splunk unless an outage has occurred, but I thought I’d check on deduplication.