Hi all,
I am running two differente environments with zeek 5.0.1 under FreeBSD 13.1. In one of the Zeek environments Zeek continuously crashed and in the other one it did not. All FreeBSD hosts are on the same version of patches and same versions of software packages.
FreeBSD release in both environments is 13.1 fully patched.
Zeek release in both environemnts is the same: 5.0.1
zkg list output in both environments:
zeek/brimsec/geoip-conn (installed: master) - Adds additional fields to the conn.log for the data obtained via Zeek’s GeoLocation feature (GeoLocation — Book of Zeek (v5.1.0)).
zeek/corelight/zeek-community-id (installed: 3.2.1) - “Community ID” flow hash support in conn.log
zeek/corelight/zeek-long-connections (installed: v1.3.1) - Find and log long-lived connections into a “conn_long” log.
zeek/salesforce/hassh (installed: master) - HASSH is used to identify specific Client and Server SSH implementations.
zeek/salesforce/ja3 (installed: master) - JA3 creates 32 character SSL client fingerprints and logs them as a field in ssl.log.
zeek/zeek/zeek-netmap (installed: v2.0.0) - Packet source plugin that provides native Netmap support.
The packages installed on the hosts in both environments are:
bash-5.1.16 GNU Project’s Bourne Again SHell
bison-3.8.2,1 Parser generator from FSF, (mostly) compatible with Yacc
ca_root_nss-3.81 Root certificate bundle from the Mozilla Project
ccache-3.7.12_3 Tool to minimize the compile time of C/C++ programs
cmake-3.23.3 Cross-platform Makefile generator
curl-7.84.0 Command line tool and library for transferring data with URLs
expat-2.4.8 XML 1.0 parser written in C
geoipupdate-4.9.0_5 Fetch the latest copies of the GeoIP2 databases
gettext-runtime-0.21 GNU gettext runtime libraries and programs
git-2.37.2 Distributed source code management tool
google-perftools-2.10_2 Fast, multi-threaded malloc() and nifty performance analysis tools
indexinfo-0.3.1 Utility to regenerate the GNU info page index
ipsumdump-1.86 Produce ASCII summary of network traffic or tcpdump(1) file
jq-1.6 Lightweight and flexible command-line JSON processor
jsoncpp-1.9.5 JSON reader and writer library for C++
lbl-cf-1.2.5 Unix time to formatted time and date filter
lbl-hf-1.9 Address to hostname filter
libarchive-3.6.1,1 Library to create and read several streaming archive formats
libedit-3.1.20210910,1 Command line editor library
libevent-2.1.12 API for executing callback functions on events or timeouts
libffi-3.4.2 Foreign Function Interface
libiconv-1.17 Character set conversion library
libidn2-2.3.3 Implementation of IDNA2008 internationalized domain names
liblz4-1.9.3,1 LZ4 compression library, lossless and very fast
libmaxminddb-1.6.0 Library for the MaxMind DB file format used for GeoIP2
libnghttp2-1.48.0 HTTP/2.0 C Library
libpsl-0.21.1_4 C library to handle the Public Suffix List
librdkafka-1.8.2 Apache Kafka C/C++ library
libssh2-1.10.0,3 Library implementing the SSH2 protocol
libtextstyle-0.21 Text styling library
libunistring-1.0 Unicode string library
libunwind-20211201_1 Generic stack unwinding library
libuv-1.44.2 Multi-platform support library with a focus on asynchronous I/O
m4-1.4.19,1 GNU M4
mpdecimal-2.5.1 C/C++ arbitrary precision decimal floating point libraries
ninja-1.10.2,2 Small build system closest in spirit to Make
oniguruma-6.9.8_1 Regular expressions library compatible with POSIX/GNU/Perl
p5-Authen-SASL-2.16_1 Perl5 module for SASL authentication
p5-CGI-4.54 Handle Common Gateway Interface requests and responses
p5-Clone-0.45 Recursively copy Perl datatypes
p5-Digest-HMAC-1.04 Perl5 interface to HMAC Message-Digest Algorithms
p5-Encode-Locale-1.05 Determine the locale encoding
p5-Error-0.17029 Error/exception handling in object-oriented programming style
p5-GSSAPI-0.28_2 Perl extension providing access to the GSSAPIv2 library
p5-HTML-Parser-3.78 Perl5 module for parsing HTML documents
p5-HTML-Tagset-3.20_1 Some useful data table in parsing HTML
p5-HTTP-Date-6.05 Conversion routines for the HTTP protocol date formats
p5-HTTP-Message-6.37 Representation of HTTP style messages
p5-IO-HTML-1.004 Open an HTML file with automatic charset detection
p5-IO-Socket-INET6-2.72_1 Perl module with object interface to AF_INET6 domain sockets
p5-IO-Socket-SSL-2.074 Perl5 interface to SSL sockets
p5-LWP-MediaTypes-6.04 Guess media type for a file or a URL
p5-Mozilla-CA-20211001 Perl extension for Mozilla CA cert bundle in PEM format
p5-Net-SSLeay-1.92 Perl5 interface to SSL
p5-Socket6-0.29 IPv6 related part of the C socket.h defines and structure manipulators
p5-TimeDate-2.33,1 Perl5 module containing a better/faster date parser for absolute dates
p5-URI-5.12 Perl5 interface to Uniform Resource Identifier (URI) references
pcre-8.45_1 Perl Compatible Regular Expressions library
pcre2-10.40 Perl Compatible Regular Expressions library, version 2
perl5-5.32.1_1 Practical Extraction and Report Language
pkg-1.18.3 Package manager
py39-backports-1 Shared namespace shim for py-backports.* ports
py39-btest-0.71 Simple driver for basic unit tests
py39-configparser-3.5.3_1,1 INI style configuration file parser
py39-gitdb-4.0.9 Git Object Database
py39-gitpython-3.1.27 Python Git Library
py39-semantic-version-2.10.0 Python library provides a few tools to handle SemVer in Python
py39-setuptools-63.1.0 Python packages installer
py39-smmap-5.0.0 Sliding-window memory map manager
py39-sqlite3-3.9.13_7 Standard Python binding to the SQLite3 library (Python 3.9)
python39-3.9.13 Interpreted object-oriented programming language
readline-8.1.2 Library for editing command lines as they are typed
restic-0.13.1_3 Fast, secure, efficient backup program
rhash-1.4.3 Utility and library for computing and checking of file hashes
rsync-3.2.4_2 Network file distribution/synchronization utility
sqlite3-3.39.0,1 SQL database engine in a C library
swig-4.0.2 Generate wrappers for calling C/C++ code from other languages
tmux-3.3a Terminal Multiplexer
xxhash-0.8.1_2 Extremely fast non-cryptographic hash algorithm
zstd-1.5.2 Fast real-time compression algorithm
Environment A consists in one FreeBSD manager and two workers. In this environment Zeek crashes randomly.
- Node.cfg in environment A:
[manager]
type=manager
host=172.22.58.5
[logger]
type=logger
host=172.22.58.5
[proxy]
type=proxy
host=172.22.58.5
[idps-prod]
type=worker
host=172.22.58.2
interface=netmap::vtnet2
aux_scripts=idps-prod.zeek
[idps-mgmt]
type=worker
host=172.22.58.3
interface=netmap::vtnet2
aux_scripts=idps-mgmt.zeek
[idps-vpn]
type=worker
host=172.22.58.2
interface=netmap::vtnet3
aux_scripts=idps-vpn.zeek
[idps-win]
type=worker
host=172.22.58.3
interface=netmap::vtnet3
aux_scripts=idps-win.zeek
Enviroment B consist in only one FreeBSD host with clustered config also. In this environment, Zeek never crashes. All works ok.
- Node.cfg in environment B:
[logger]
type=logger
host=localhost
[manager]
type=manager
host=localhost
[proxy]
type=proxy
host=localhost
[nsm-honeypot]
type=worker
host=localhost
interface=netmap::vtnet3
aux_scripts=honeypot.zeek
I must point out that in both environments, with the same configuration, same software versions, etc., Zeek 4.0.X LTS works without problems …
Crash info:
Zeek 5.0.1
FreeBSD 13.1-STABLE
Zeek plugins:
Corelight::CommunityID - “Community ID” flow hash support in the connection log (dynamic, version 3.2.0)
Zeek::Netmap - Packet acquisition via Netmap (dynamic, version 1.0.0)
==== No reporter.log
==== stderr.log
listening on vtnet3
[broker/ERROR] 2022-08-28T16:33:43.303 unable to find a master for zeek/known/hosts
[broker/ERROR] 2022-08-28T16:33:43.303 unable to find a master for zeek/known/services
[broker/ERROR] 2022-08-28T16:33:43.303 unable to find a master for zeek/known/certs
/opt/zeek/share/zeekctl/scripts/run-zeek: line 110: 833 Segmentation fault nohup “$myzeek” “$@”
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) 33554432
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-i netmap::vtnet3 -U .status -p zeekctl -p zeekctl-live -p local -p idps-win /opt/zeek/etc/site/startup.zeek zeekctl base/frameworks/cluster zeekctl/auto idps-win.zeek -C
==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
ZEEKPATH=/nsm/zeek/spool/installed-scripts-do-not-touch/site::/nsm/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/b
uiltin-plugins
CLUSTER_NODE=idps-win
==== .status
RUNNING [run_loop]
Any idea?