Modify log rotation

Zeek
1

Hello,

I’m trying to change the way zeek rotates logs.
My logs are collected by fluentd with the tail input. If fluentd resumes reading a log after a pose and after a log rotation, it needs the log to have the same inode to be able to resume from the current position.
So the archived log must only be renamed and remains in the same directory without compression.

Is there a way to do that?

Thanks
Romain

2

Hi there and welcome!

Yes, that should be possible — I haven’t used fluentd but that’s basic log rotation in the same directory, without any post-processing or archival. Details will depend on how you’re running Zeek — in a standalone setting, via zeekctl, or the management framework. You basically want a logging setup in which:

  • Log::default_rotation_interval specifies the interval
  • Log::default_rotation_dir remains ""
  • Log::rotation_format_func is much like the logging framework’s default (see here)
  • Log::default_rotation_postprocessor_cmd remains "".

If you run Zeek via zeekctl or the management framework, you’ll need to tweak things a bit to get to that constellation, since they both do more than you need. With Zeek running stand-alone as single process, the logging framework’s default configuration is very close to what you need — you just need to specify a log rotation interval. For example, if I run this …

$ zeek -r large.pcap Log::default_rotation_interval=10mins

… then I get logs that look like this:

$ ls -la
...
-rw-r--r--. 1 christian christian   1080 Apr 12 18:15 conn.2009-11-18-00-01-33.log
-rw-r--r--. 1 christian christian   1558 Apr 12 18:15 conn.2009-11-18-00-10-03.log
-rw-r--r--. 1 christian christian   2106 Apr 12 18:15 conn.2009-11-18-00-20-03.log
...

… in the local directory.

Something will need to delete the rotated logs once fluentd has ingested them — not sure if you already have a solution for that.

I suggest you take a look at the logging framework’s rotation docs, and in case you’re using zeekctl, its default logging configuration.

Let us know how it goes — happy to help further once you’ve explored this a bit.

Best,
Christian

3

Yes, I have a solution that can delete logs after fluentd has ingested them. But still struggling in modifying zeek log rotation.

I use zeekctl and I’ve tryed to redef a few things in local.zeek but it seems to me that it is already the default setup.
Here is my local.zeek that does not seem to do anything

redef digest_salt = "Please change this value.";
@load misc/loaded-scripts
@load tuning/defaults
@load misc/capture-loss
@load misc/stats
@load frameworks/software/vulnerable
@load frameworks/software/version-changes
@load-sigs frameworks/signatures/detect-windows-shells
@load protocols/ftp/software
@load protocols/smtp/software
@load protocols/ssh/software
@load protocols/http/software
@load protocols/dns/detect-external-names
@load protocols/ftp/detect
@load protocols/conn/known-hosts
@load protocols/conn/known-services
@load protocols/ssl/known-certs
@load protocols/ssl/validate-certs
@load protocols/ssl/log-hostcerts-only
@load protocols/ssh/geo-data
@load protocols/ssh/detect-bruteforcing
@load protocols/ssh/interesting-hostnames
@load protocols/http/detect-sqli
@load frameworks/files/hash-all-files
@load frameworks/files/detect-MHR
@load policy/frameworks/notice/extend-email/hostnames
@load packages
@load policy/tuning/json-logs.zeek
@load policy/protocols/conn/mac-logging.zeek

redef LogAscii::gzip_level = 0;
redef Log::default_rotation_dir = "";
redef Log::default_rotation_postprocessor_cmd = "";

module Log;

function my_rotation_format_func(ri: Log::RotationFmtInfo): Log::RotationPath
        {
        local rval: Log::RotationPath;
        local open_str: string;

        # The reason for branching here is historical:
        # the default format path before the intro of Log::rotation_format_func
        # always separated the path from open-time using a '-', but ASCII's
        # default postprocessor chose to rename using a '.' separator.  It also
        # chose a different date format.
        if ( ri$postprocessor == __default_rotation_postprocessor &&
            ri$writer == WRITER_ASCII &&
            ri$writer in default_rotation_postprocessors &&
            default_rotation_postprocessors[WRITER_ASCII] == default_ascii_rotation_postprocessor_func)
                {
                open_str = strftime(Log::default_rotation_date_format, ri$open);
                rval = RotationPath($file_basename=fmt("%s.%s", ri$path, open_str));
                }
        else
                {
                open_str = strftime("%y-%m-%d_%H.%M.%S", ri$open);
                rval = RotationPath($file_basename=fmt("%s-%s", ri$path, open_str));
                }

        return rval;
        }
redef Log::rotation_format_func = my_rotation_format_func;
4

Mostly as addition/pointer, there’s the json-streaming-logs package that was made for exactly that use-case. It rotates into conn.log, conn.log.1, conn.log.2, conn.log.3 with a maximum number on the files to keep.

Might be easier to leverage that and/or if it doesn’t fully work out, patch and contribute upstream.

From the docs/scripts:

Files are rotated in the current log directory without being compressed so that any log shipper has a while to catch up before the log file is deleted.

 	## If rotation is enabled, this is the number of extra files that Zeek will 
	## leave laying around so that any process watching the inode can finish.  
	## The files will be named with the following scheme: `json_streaming_<path>.<num>.log`.
	## So, the first conn log would be named: `json_streaming_conn.1.log`.
	const JSONStreaming::extra_files: int = 4 &redef;

Hope this helps,
Arne

5

Awesome, that does the job perfectly :ok_hand:
Thanks a lot!