You may try experimenting with Flume. It’s part of that larger group of Hadoop related Apache projects, and included in Cloudera’s distro.
It’s quite a capable log management tool, and can provide a lot of flexibility.
Hi Ben,
We do use Flume for applications we can’t extend with Kafka. There are pluses and minuses for either route but with Bro we wanted to queue directly from the output process code to avoid the additional IO of disk read/writes. Adding another processor like Flume also increases the complexity and potential failure points.
You do give up some reliability of message delivery but it’s an acceptable risk for us at the moment. A good Kafka/Zookeeper environment is over-engineered anyway and Bro already is a single point of failure.