logger large memory usage

Zeek
1

Hey all,

Im running a bro cluster in 10G network, writing logs to redis server directly use plugins, when the cluster is running, the memory usage of the logger has been growing larger and larger(hundreds of Gb), seems like the logger cannot handle the huge number of the log messages, so the question is whats the processing capacity of the logger? In my case, 25000 msgs/sec to redis. And why not use more loggers in one cluster but only one?

Any insight would be helpful.

Bowen Li

2

I might be wrong but using Redis you can’t store more than your actual RAM capacity
Check out: https://redis.io/topics/faq

Hope this helps,
Tomas

3

Multiple loggers is something that is being worked on. broctl git repository has initial support for running more than one worker on a cluster. It doesn't really work right if you are logging to files, but if you are using kafka or redis to aggregate logs it will work fine.