Sorry beginner question here:
But I know you can ingest logs into Splunk, and Elastic Search.
So I know SecurityOnion has an ELK stack and it looks like they get sent right to Logstash - ES - Kibana
RockNSM looks almost the same but it has a stop off at Kafka before forwarding to Logstash.
Trying to figure out is there a benefit for Kafka.
Also looking at using Splunk instead of ES.
I know I can use the TA and monitor the logs from splunk, but would it be better to monitor from Kafka?
I guess I need to understand more of how Kafka fits.
Thanks
Dave
David,
Think of Kafka as a message queue between Zeek and ELK.
I think it depends partly on the scale of your setup. On a small system, it’s probably not needed, but as you scale up it becomes much more useful. I ran a Zeek system with ELK for a few years before moving from security to networking. My ELK system ingested about 3 billion messages a week. I used Redis between Zeek and ELK. Redis was just a message queue. It provided a buffer between Zeek and ELK. That helped smooth out bursts of log messages from Zeek since it could generate messages a lot faster than my ELK cluster could process them. It also meant I could take the ELK cluster down for maintenance without stopping Zeek. Messages would just queue up in Redis until I brought ELK back up.
Once I found Kafka I had planned to replace Redis with it, but I was offered the networking gig before I made that happen. Kafka had the benefits of being properly clustered itself and wrote to disk instead of RAM which was the main limitation of using Redis. Kafka also had the feature of being able to support multiple independent consumers so that I could have data feeding into multiple systems if I wanted to.
the biggest reason is absorbing back pressure from logstash or other ingesting tools. In past the back pressure from logstash would cause CPU spikes on the originating endpoint.
second, we can write programs to clean, modify and enrich data before throwing at the ingesting tools making our log processing pipelines indipendedent. Giving us flexibility of migrating from logstash to Humio or Splunk and not worry about wasting all the efforts you put into logstash pipelines.
There are some good patterns here. We observed that it helps a lot to just ship logs from each NSM sensor as soon as data is collected, with minimal, if any, processing. That’s why we ship logs (with syslog-ng) to a RabbitMQ instance, via AMQPS. No extra processing is done there, it’s just buffering. We then have a set of python workers fetching messages from Rabbit and doing all of the processing.
No Kafka here but just simple solutions and avoiding any processing on endpoints.
also it’s a good security practice to offload logs from endpoints ASAP. During incidnet endpoints can go offline or the data can be impacted especially if the endpoints are compromised.
Hi Dave,
To clarify, Security Onion may also include redis in the pipeline, depending on what kind of architecture you are deploying. For more information, please see:
https://securityonion.readthedocs.io/en/latest/architecture.html#distributed
Hope that helps!