I am testing Zeek-4.1.0 in preparation for an upgrade in production (from Zeek-4.0.2). I compiled the RPM myself (using --disable-btest --build-static-broker switches like I had done for 4.0.2).
I have a test setup on a CentOS7 machine where Zeek-4.0.2 was already running. On upgrading the RPM to 4.1.0 and restarting zeek service all the nodes crashed and the following was logged to the stderr.log file
internal error: type inconsistency in ZVal constructor
/opt/zeek/share/zeekctl/scripts/run-zeek: line 110: 11357 Aborted (core dumped) nohup “$myzeek” “$@”
I am able to run zeek against PCAPs (-r pcap) but not able to start it as a service.
Anybody has any ideas about probable causes and/or fixes.
I am testing Zeek-4.1.0 in preparation for an upgrade in production (from
Zeek-4.0.2). I compiled the RPM myself (using --disable-btest
–build-static-broker switches like I had done for 4.0.2).
The first thing that comes to mind is to leave off the —disable-btest and after building zeek, do:
cd testing/btest
make
to see if it reports any problems. Note, it will take quite a while to run through the tests.
On further investigation, it seems that zeek only crashes when I do a zeekctl status.
If I don’t fire that command, there is no problem. A zeekctl status takes a long time & reports all nodes as crashed. The error message also appears in stderr only after that.
I run Zeek using pf_ring and command to run zeekctl is
And this is the backtrace in the .crash-diag.out file
Thread 1 (Thread 0x7f103e4de3c0 (LWP 7332)): #0 0x00007f103c031387 in raise () from /lib64/libc.so.6 #1 0x00007f103c032a78 in abort () from /lib64/libc.so.6 #2 0x00000000008d97aa in zeek::Reporter::InternalError (this=, fmt=fmt@entry=0xf90d40 “type inconsistency in ZVal constructor”) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Reporter.cc:216 #3 0x0000000000948ef2 in zeek::ZVal::ZVal (this=0x7fffdbe91a98, v=…, t=…) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/ZVal.cc:33 #4 0x000000000092d3f6 in zeek::RecordVal::Assign (this=this@entry=0x6153510, field=field@entry=0, new_val=…) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Val.cc:2916 #5 0x00000000008a1b34 in zeek::BifFunc::Broker::__peers_bif (frame=, BiF_ARGS=) at comm.bif:120 #6 0x00000000008a68d0 in zeek::detail::BuiltinFunc::Invoke (this=0x33c6cd0, args=0x7fffdbe91d50, parent=0x61666a0) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Func.cc:796 #7 0x000000000086b2cc in zeek::detail::CallExpr::Eval (this=0x2fb2460, f=0x61666a0) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Expr.cc:4501 #8 0x000000000090d4aa in zeek::detail::ReturnStmt::Exec (this=, f=, flow=) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Stmt.cc:1581 #9 0x000000000090f022 in zeek::detail::StmtList::Exec (this=, f=0x61666a0, flow=@0x7fffdbe91e14: zeek::detail::FLOW_RETURN) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Stmt.cc:1630 #10 0x00000000008af759 in zeek::detail::ScriptFunc::Invoke (this=0x2fb2520, args=0x7fffdbe91fc0, parent=0x6154300) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Func.cc:430 #11 0x000000000086b2cc in zeek::detail::CallExpr::Eval (this=0x5c09130, f=0x6154300) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Expr.cc:4501 #12 0x000000000086c8cc in zeek::detail::AssignExpr::Eval (this=0x5c09210, f=0x6154300) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Expr.cc:2554 #13 0x000000000090e4bc in zeek::detail::ExprStmt::Exec (this=0x5c092f0, f=0x6154300, flow=@0x7fffdbe92134: zeek::detail::FLOW_NEXT) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Stmt.cc:405 #14 0x000000000090f022 in zeek::detail::StmtList::Exec (this=, f=0x6154300, flow=@0x7fffdbe92134: zeek::detail::FLOW_NEXT) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Stmt.cc:1630 #15 0x000000000090f022 in zeek::detail::StmtList::Exec (this=, f=0x6154300, flow=@0x7fffdbe92134: zeek::detail::FLOW_NEXT) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Stmt.cc:1630 #16 0x00000000008af759 in zeek::detail::ScriptFunc::Invoke (this=0x3c18610, args=0x616aa80, parent=0x0) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Func.cc:430 #17 0x000000000085437e in zeek::EventHandler::Call (this=, vl=vl@entry=0x616aa80, no_remote=) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/EventHandler.cc:107 #18 0x00000000008535f6 in zeek::Event::Dispatch (this=this@entry=0x616aa60, no_remote=, no_remote@entry=false) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Event.cc:60 #19 0x0000000000853b44 in zeek::EventMgr::Drain (this=0x13fdd20 zeek::event_mgr) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/Event.cc:162 #20 0x00000000009019a3 in zeek::run_state::detail::run_loop () at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/RunState.cc:330 #21 0x0000000000722180 in main (argc=, argv=) at /home/admin/rpmbuild/BUILD/zeek-4.1.0/src/main.cc:59
# Show all output of the zeekctl status command. If set to 1, then all output
# is shown. If set to 0, then zeekctl status will not collect or show the peer
# information (and the command will run faster).
StatusCmdShowAll = 0
the status command only uses broker if you have changed that to 1.
Changing the option StatusCmdShowAll to 0 works. The status command no longer crashes everything. Although the behavior of peerstatus does not change and its use will still crash everything (probably because it goes through broker). But it is not a command that is often used by me.
I wonder if this is a broker bug or is something broken in my build workflow that causes this error.
I wonder if this is a broker bug or is something broken in my build
workflow that causes this error.
It’s definitely an internal bug - there shouldn’t be anything a user can do with their workflow that triggers an internal error like that. FWIW, the most likely cause is a mismatch between the type of an object when presented to broker vs. the type the receiver expects (so not a broker bug, but a broker usage bug).