Hello everyone,
If you followed last year’s Zeek Week, you might be aware that we have been working on a new way to more easily distribute Zeek Table content in a cluster setup. We now have a working prototype - and I would be happy for feedback if someone wants to start playing with it.
We tried to make this feature as easy to use as possible. In the case that you just want to distribute a table over an entire Zeek-cluster, you only have to add &backend=Broker::MEMORY to the table definition.
So - for example:
global table_to_share: table[string] of count &backend=Broker::MEMORY;
This will automatically synchronized the table over the entire cluster. In the background, a Broker store (in this case a memory-backed store) is created and used for the actual data synchronization. Changes to the table are automatically sent to the broker store and distributed over the cluster.
We also support persistent broker stores. At the moment you need to specify the path in which the database should be stored for this feature. Example:
redef Broker::auto_store_db_directory = "[path]";
global table_to_share: table[string] of count &backend=Broker::SQLITE;
Data that is stored in the table will be persistent across restarts of Zeek.
Current limitations:
* there is no conflict resolution. Simultaneous inserts for the same key will probably lead to a divergent state over the cluster. This is by design - if you need to be absolutely sure that you do not loose any data, or if you want conflict resolution for multiple inserts, you will still have to roll your own script-level logic using events.
* tables only can have a single index, multi-indexed tables (like table[string, count] of X) are not yet supported
* tables only can have simple values. Tables that store records, tables, sets, vectors are not supported. The reason for this is that we cannot track table-changes in these cases.
* &expire_func cannot be used simultaneously. Normal expiry should work correctly.
* documentation is basically still completely missing - I will write it over the next days.
If you want to try this you have to compile the topic/johanna/table-changes branch of the Zeek repository. To check out this branch into a new directory, use something like:
git clone GitHub - zeek/zeek: Zeek is a powerful network analysis framework that is much different from the typical IDS you may know. --branch topic/johanna/table-changes --recursive [target-directory]
Please let me know if you have any feedback/questions/problems 
Johanna