We’re proud to announce the release of Zeek 9.0! This long-term support release builds on our work in 8.1 and 8.2, and provides important new ways to communicate events in the cluster, consume encapsulated traffic, and orchestrate cluster processes. It also features a range of protocol analyzer improvements and additions, including a new version of our Spicy protocol parser generator.
Global Eventing in the Cluster via ZeroMQ
Zeek 9 is the first LTS release to power its cluster communication via the ZeroMQ messaging library by default. This switch, first introduced in Zeek 8.1, simplifies cluster communication by establishing a centralized physical communication topology over which Zeek conducts topic-based publish/subscribe messaging:
For script authors, this change is a liberation. Where nodes previously had to “ping-pong” events via other nodes and topics to communicate with the desired destination, they can now simply publish directly. For example, worker-to-worker eventing is now immediate. Publishing an event to all nodes in the cluster is now feasible for the first time, and it’s now possible to tap the entirety of a cluster’s event communication in a single spot.
As with Broker, the cluster functions out-of-the box, without an external broker service. (ZeroMQ power users can choose to run their own, though.) The Broker cluster backend remains available in all builds, and is easily re-enabled for users running legacy scripts or Zeek packages that require Broker’s eventing. Broker itself is now deprecated in its entirety.
Native Packet Ingestion for Encapsulated Traffic
Zeek 9 ships with a new packet source that works quite differently from its packet-sniffing cousins. Aimed at processing all-encapsulated traffic in cloud environments, it builds on the pluggable flow-tuple functionality we introduced in Zeek 8 and turns the Zeek cluster into a UDP server that can process encapsulated packet streams directly and scalably — no need for packet-sniffing workarounds that cause all traffic to get logged with encapsulation.
If you follow our blog, you’ve seen us write about this functionality. Please refer to the package source’s documentation for more information on supported encapsulations and usage.
Modern Cluster Process Orchestration on Linux
Zeek now ships with mature support for systemd-orchestrated clusters on Linux. Zeek’s systemd generator, informed by a new, declarative zeek.conf cluster configuration file, produces the systemd unit files necessary to launch the cluster. It brings all of the flexibility afforded by systemd to Zeek cluster management, including process resource & capability management, launch ordering & dependencies, a cluster structured into systemd slices, and support for customization via drop-in units. Multi-machine clusters are supported too: a single zeek.conf file can define a cluster that spans multiple machines, each running a locally generated, systemd-managed subset.
As a beneficial consequence, this approach cleanly separates cluster orchestration from Zeek configuration and deployment. Systemd manages the processes, while you install and configure Zeek as usual via packages, redefs, your own scripting, etc.
Our traditional cluster management tool, zeekctl, remains supported and functional.
Improved Protocol Analysis
As usual, Zeek’s protocol parsers have received a wide range of improvements. The LDAP analyzer now forwards SASL payloads to GSSAPI and NTLM parsers for enhanced logging into ntlm.log and kerberos.log, the Syslog analyzer now supports TCP, and multicast DNS transactions are now logged as separate transactions since they occur on separate network flows. The TLS analyzer has gained a new ssl_extension_encrypted_client_hello event to report on ECH, the HTTP analyzer features several new weirds to report unusual behavior and no longer reports the QUERY method (RFC 10008) as weird, the FTP analyzer now also logs sessions that only feature login and logout, and a new, optional ZIP file analyzer reports contained files to files.log.
Many analyzers have received additional tuning knobs to safeguard against abusive traffic. These should not matter in normal network traffic, and most trigger an “X” character in the conn.log entries’ history field to indicate that a threshold prevented further analysis. A corresponding weird.log entry then explains the specific threshold.
Log Schema Updates
Here’s how Zeek 9’s analyzer improvements change the structure of its logs.
Compared to 8.2.x:
-
In NTP analysis, the new logs ntp_control and ntp_private report on NTP modes 6 and 7, respectively.
-
With the protocols/conn/multicast-participants.zeek policy script, a new multicast_participants.log reports on the multicast speakers on your network based on IGMP activity on the wire.
-
With the protocols/ssl/ssl-log-ext.zeek policy script, ssl.log now gains two fields,
ech_kdf_idandech_aead_id, reporting the HPKE KDF and AEAD identifiers from the TLS 1.3 ECH extension.
Compared to 8.0.x the above apply, as well as the following:
-
dns.log now features
opcodeandopcode_namefields to support DNS notifications (RFC 1996). The protocols/dns/disable-opcode-log-fields.zeek policy script removes these, if desired. -
In ssh.log, the
host_keyfingerprint field was replaced with a newhost_key_fingerprintfield. Part of our gradual MD5 phase-out, the new field uses the same format asssh-keygen -l, with SHA256 for hashing. The protocols/ssh/md5-host-key-logging.zeek policy script restores the old field, if desired. -
Similarly, in kerberos.log, the protocols/krb/ticket-logging.zeek policy script now adds fields
auth_ticket_sha256andnew_ticket_sha256instead ofauth_ticketandnew_ticket, which held MD5 hashes. The protocols/krb/md5-ticket-logging.zeek policy script brings back the original fields, if desired.
These comparisons are produced with help of the logschema Zeek package.
Spicy Updates
Zeek 9 ships with Spicy 1.17.0, which builds on the updates we shipped in Zeek 8.1 and 8.2 along two major themes.
First, Spicy’s runtime performance got some significant improvements. With the introduction of a control flow graph, the generated code is able to perform more optimizations. These include dead code elimination, constant propagation, removing unused fields, moving last uses, propagating function return values, and more. Microbenchmarks show about a 20-60% performance improvement. Release builds also no longer carry debugging information. Depending on the parser, that may be less (if you use all of Spicy’s features, there’s less ability to optimize).
Second, with the help of Maor Hamami, we have also added Windows (via MSVC) support to Spicy. This is still not a supported platform, but CI coverage should ensure continued compatibility.
Other Additions
In the scripting language, the new ?as operator allows checking whether the lefthand operand can be safely converted to the righthand type. Users upgrading from 8.0 or earlier should also review the &publish_on_change attribute, introduced in 8.2, that automatically distributes updates to tables and sets across the cluster.
In JSON logging, users now have multiple options to configure Zeek’s handling of binary data in the logs, resolving a long-standing ambiguity. The software framework features improved control over version findings via the new Software::parse_cache_interval, Software::found_cache_interval, and Software::max_software_cache_size tunables.
We also developed many security fixes and infrastructure to make sure Zeek remains fully secure. This doesn’t have any pertinent impact on 9.0 in particular, but previous and future releases leverage the full power of the scanning efforts and new infrastructure that we have access to.
The full set of changes is far larger than we can capture here, particularly if you’re upgrading from Zeek 8. Please review Zeek’s and Spicy’s release notes for the full list of changes, and consult our documentation.
As our latest LTS release, the Zeek 9.0.x series will see security fixes and relevant backports for a bit over a year. Our past LTS release, Zeek 8.0.x, will stop receiving updates when we release 9.1 in about four months.
We encourage all users to upgrade to Zeek 9 at this time.
Thanks to our contributors!
Our work on Zeek 9 began in August and covers some 1,300 commits in nearly 400 pull requests. We’re particularly grateful to our community members who contributed to this release — thank you!
Aaron J. Scantlin (@scantlina), @airshelley, Alexey Rubanik (@Rubanik-Alexei), Azil (@achill06), Bauti Peirone (@bautipeirone), Blair McCormick (@blairSmcc03), Craig Leres (@leres), Craig P (@detection-labs), @Cxiyuan, Dragon Roar (@lczllx), @eb-anssi, Eduardo Kohn (@eagle-head), Eldon Koyle (@ekoyle), @franky-m, Jan Grashöfer (@J-Gras), Javid Khan (@dxbjavid), Jie (Link) Zheng (@chillancezen), @jmestwa-coder, Julio César Suástegui (@juliosuas), Keith J. Jones (@keithjjones), Kevin Harrington (@ke5C2Fin), Maor Hamami (@mamaorha), Mario D (@mari0d), Matthias Vallentin (@mavam), Mohan Dhawan (@mdhawan), Mohan Dhawan (@Mohan-Dhawan), neil (@neilpang), Otto Fowler (@ottobackwards), Peter Cullen (@pbcullen), Seth Grover (@mmguero), Shubham Kumar (@chmodshubham), Simeon Miteff (@simeonmiteff), Smoot (@stevesmoot), Stefan Götz (@stefangoetzcorelight), Steven (@malvidin), Sujala Vasanthasena Nelavai (@sujalavnelavai), Swastik Bose (@VasuBhakt), @tim-cyb, Urval (@urvalkheni), @uwezkhan and Yacin Nadji (@ynadji) for their contributions to this release.
You too can be in this list! Are you interested in contributing to Zeek but unsure how? Here are 5 ways to get started. In our Zeek community call, on the first Wednesday of each month, we regularly report on recent developments in the Zeek community. We encourage you to join and ask questions! These calls are also available for later viewing on our YouTube channel.
As always, we also thank Corelight for its continued support of the Zeek project.