Send logs from zeek to redis i.e. a Redis log writer

Zeek
1

Hi, I am trying to create a plugin which sends logs from zeek to redis database. Already we have many plugins available on the internet which sends logs from zeek to kafka, but i couldnt find anything to that on redis. I have shared my work below. Can anyone help me with this task?

redis-writer.zeek:

module Redis;

export {
  ## Configuration options for the Redis writer.
  ##
  ## To use this writer, add the following to your script:
  ##
  ##     @load aux/redis-writer
  ##
  ##     redef Redis::servers = {
  ##         ["redis://127.0.0.1:6379"],
  ##     };
  ##
  ##     Log::add_writer(REDIS_LOG, [$writer=REDIS_WRITER]);
  ##
  ##     redef Log::default_writer = REDIS_LOG;
  ##
  ## The default log stream will then be written to Redis.
  ##
  redef record Redis::options += {
    ## The Redis server(s) to write to. This should be a set of Redis URIs,
    ## where each URI is a string of the form "redis://<hostname>:<port>".
    servers: set[string] = [
      "redis://127.0.0.1:6379"
    ],

    ## The Redis key prefix to use when writing logs. This allows you to
    ## namespace your logs in Redis.
    key_prefix: string = ""
  };
}

export {
  ## //Constants for the Redis writer.
  const REDIS_WRITER: Log::WriterInfo = Log::WriterInfo($name="Redis::WRITER", $priority=5);
  const REDIS_LOG: Log::Log = Log::Log([$columns=Log::default_columns,
                                         $name="redis", $id=Log::default_id,
                                         $path="", $writer=REDIS_WRITER]);
}

type RedisWriter: Log::FilterWriter &optionals += {
  ## //The Redis server(s) to write to. This should be a set of Redis URIs,
  ## //where each URI is a string of the form "redis://<hostname>:<port>".
  servers: set[string] &optional;

  ## //The Redis key prefix to use when writing logs. This allows you to
  ## //namespace your logs in Redis.
  key_prefix: string &default="";

  ## //The Redis connection pool to use. This is an internal option and should
  ## //not be set by the user.
  conn_pool: Redis::ConnectionPool &default=Redis::ConnectionPool(Redis::options$servers);
};

global redis_writer: RedisWriter;

event bro_init()
{
  if ( ! redis_writer?$writer )
    redis_writer$writer = Log::WRITER_ASYNC;

  if ( ! redis_writer?$filters )
    redis_writer$filters = Log::default_filters;

  if ( ! redis_writer?$prefix )
    redis_writer$prefix = Redis::options$key_prefix;

  if ( ! redis_writer?$servers )
    redis_writer$servers = Redis::options$servers;

  if ( ! redis_writer?$conn_pool )
    redis_writer$conn_pool = Redis::ConnectionPool(redis_writer$servers);

  Log::add_filter_writer(REDIS_WRITER, [$writer=redis_writer]);
}

event bro_done()
{
  redis_writer$conn_pool$terminate();
}

function bro_redis_write(log: Log::Info, writer: RedisWriter): bool
{
  local Redis::Connection conn = writer$conn_pool$acquire();
  local string key = fmt("%s%s:%s", writer$prefix, log$path, log$id);

  conn$publish(key, log$to_json(), conn$reconnect_on_error);
  writer$conn_pool$release(conn);

  return T;
}

lines i added in locak.zeek:

@load redis-writer
redef Redis::servers = {
  ["redis://127.0.0.1:6379"],
};
Log::add_writer(REDIS_LOG, [$writer=REDIS_WRITER]);
redef Log::default_writer = REDIS_LOG;
2

Hi there,

You’ll need to build a Zeek plugin to implement the log writes to Redis, via a new log writer. I’m afraid that’s beyond the scope of what we can hammer out jointly here in Discourse, but there is an existing Zeek package to help you get started — take a look at GitHub - sedarasecurity/zeek-redis: Redis writer for Zeek . I’m not sure what’s the state of the project, but you could fork or contact its authors.

Best,
Christian

3

Tiny promotion: If you’re just testing things out, brave, or in an experimental/prototype/proof-of-concept/low-traffic environment, the following gives you a Redis exporter if you have JavaScript enabled in Zeek:

/* Redis exporter using Log::log_stream_policy hook and Redis's pub/sub */
const redis = require('redis');

BigInt.prototype.toJSON = function() {
  return this.toString();
}

const client = redis.createClient();
client.connect();

zeek.hook('Log::log_stream_policy', (rec, stream_id) => {
  if ( ! client.isReady )  // Short-cut: not yet connected, just drop the record.
    return;

  let log_rec = zeek.select_fields(rec, zeek.ATTR_LOG);
  let data = JSON.stringify(log_rec);
  client.publish(stream_id, data);
});

And with redis-cli, doing a SUBSCRIBE Conn::LOG gives you a stream of conn logs.

This has no error handling or configurability and might not be performant enough for production, but if you’re just exploring things, might be good enough. And there’s always the option to implement it in C++ as a plugin.