Hello,
when deploying Zeek with zeekctl and configuring multiple loggers on a single host, the current default behavior is subpar: Individual loggers will rotate their log files into the same destination directory using the same filenames and overwrite each others logs causing data loss. The same applies when using the newer supervisor / zeek-archiver setups, too, but seems ZeekControl is more prevalent in usage for now.
We’re trying to get a sense if that affects many deployments and/or how you’ve dealt with it. Below is a small poll - feel free to select which:
- We’re not using multiple loggers, our log volume isn’t high enough
- We’re using multiple loggers, but aren’t using ASCII logs (e.g. logs are streamed from Zeek to Kafka or Elasticsearch, … directly)
- We’re using multiple loggers and ASCII logs, but don’t archive logs (e.g. using json-streaming-logs and log shippers to move logs off the system to elsewhere)
- We’re using multiple loggers and fixed it via custom Zeek scripts or patching ZeekControl’s
archive-log - We’re not using ZeekControl at all and have build our own setup
- We weren’t aware there’s an issue
- I’m using a zeek-archiver setup instead of ZeekControl
0 voters
One idea currently is to suffix the filenames in the archive directory with the name of the logger that produced it and avoid the collision that way. If you’ve run into the issue and have fixed it yourself or opinions how it should be done, feel free to provide some more details as a reply in this thread or on below GithHub issue or PR.
Zeek side GitHub issue: Multi-logger cluster support for ASCII logs · Issue #2728 · zeek/zeek · GitHub
Draft PR for suffixing log files for ZeekControl: Multi-logger handling by awelzel · Pull Request #56 · zeek/zeekctl · GitHub
Thanks,
Arne