zeekctl netstats returns time out

Hi all,

Every time I run “zeekctl netstats” returns time out under FreeBSD 12.1 hosts using netmap:

root@fbsdzeek01:/nsm/zeek/logs/current # zeekctl netstats

Warning: ZeekControl plugin uses legacy BroControl API. Use

‘import ZeekControl.plugin’ instead of ‘import BroControl.plugin’

zeek: <error: time-out>

This behavior occurs in both standalone and cluster configurations. Any idea? Maybe is it a bug?

Zeek 3.0.1's `zeekctl netstats` is working for me in FreeBSD 12.1.
TCP connectivity is required for that command to work and you can read
more about the ports involved for further troubleshooting here:

    https://github.com/zeek/zeekctl#zeek-communication

If the Zeek processes are particularly busy, that could also be a
reason for timing out. The `CommTimeout` (default 10 seconds) can be
increased in `zeekctl.cfg` in that case.

- Jon

Many thanks Jon. Regarding TCP connectivity, I have neither ipfw nor pf enabled between manager and workers. And respecting to "busy" system, shouldn't be the problem either. For example, my top output in standalone config:

last pid: 6492; load averages: 0.16, 0.22, 0.22 up 0+06:21:48 22:20:43
44 threads: 1 running, 43 sleeping
CPU: 0.0% user, 0.0% nice, 1.9% system, 0.0% interrupt, 98.1% idle
Mem: 51M Active, 58M Inact, 679M Wired, 271M Buf, 5137M Free
Swap: 4096M Total, 4096M Free

  PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
45091 root 22 0 460M 111M select 1 18:29 4.71% zeek{zeek}
6492 root 20 0 1044M 4144K CPU0 0 0:00 0.05% top
45091 root 20 0 460M 111M uwait 0 0:22 0.02% zeek{caf.clock}
39952 _ntp 20 -20 1038M 4000K select 1 0:03 0.01% ntpd
45407 root 20 0 1044M 9912K select 1 0:00 0.01% sshd
45091 root 20 0 460M 111M uwait 1 0:09 0.01% zeek{caf.multiplexer}
45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.ntp/Log::WRITER_}
45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.files/Log::WRITE}
45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.capture_loss/Log}
45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.dns/Log::WRITER_}
45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.ssl/Log::WRITER_}
45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.http/Log::WRITER}
45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.loaded_scripts/L}
45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.packet_filter/Lo}
45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.stats/Log::WRITE}
45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.conn/Log::WRITER}
45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.software/Log::WR}
45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.known_services/L}
45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.x509/Log::WRITER}
45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.notice/Log::WRIT}
45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.ssh/Log::WRITER_}
45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.kerberos/Log::WR}
45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.broker/Log::WRIT}
45091 root 20 0 460M 111M uwait 1 0:01 0.00% zeek{zk.weird/Log::WRITE}
45091 root 20 0 460M 111M uwait 0 0:00 0.00% zeek{zk.dhcp/Log::WRITER}
45091 root 20 0 460M 111M uwait 1 0:00 0.00% zeek{zk.known_certs/Log:}
45091 root 20 0 460M 111M uwait 0 0:01 0.00% zeek{zk.known_hosts/Log:}
96485 root 20 0 17M 6920K select 0 0:00 0.00% sendmail
45091 root 20 0 460M 111M select 0 0:00 0.00% zeek{caf.multiplexer}

Any idea about how to debug this error?

Regards,
C. L. Martinez

Since this is working in my own environment, we could maybe compare
configs until we find the differences. What's the node.cfg you use?
If it's all just a single node using localhost, these are some of the
first things that come to mind for troubleshooting:

Confirm TCP connectivity:

# nc -zv 127.0.0.1 47761
Connection to localhost 47761 port [tcp/*] succeeded!

There's also the other 47761+ ports to try, but likely all get the
same result as the first one. An IPv4 vs. IPv6 config issue might
also be a problem and can try variations of "::1" and "localhost" in
place of "127.0.0.1" if it's all one node. To really get all IPv4,
think you can even set 127.0.0.1 in node.cfg and run like this:

    ZEEK_DEFAULT_LISTEN_ADDRESS=127.0.0.1 /usr/local/zeek/bin/zeekctl deploy

The high-level connection attempts are also logged here:

    /usr/local/zeek/logs/current/broker.log

See anything interesting there? It should have several initial
"peer-added" and "handshake successful" entries for the initial
cluster setup and then for each time you try something like `zeekctl
netstats worker-1` it will have a pair of "peer-added" and
"connection-terminated" entries.

- Jon

Hi Carlos,

As Jon said, let's check network things.

Could you confirm the ip/port used on your side ?

# netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address
(state)
tcp4 0 0 10.0.1.1.47760 *.* LISTEN

Cheers.

Jean-Philippe.

Good morning,

Many thanks for your help Jon. All my config that you have requested.

- node.cfg:

[manager]
type=manager
host=localhost

[logger]
type=logger
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=netmap:vtnet2

[worker-2]
type=worker
host=localhost
interface=netmap:vtnet3

- sockstat -l4:
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root sendmail 50520 3 tcp4 127.0.0.1:25 *:*
root sshd 31667 3 tcp4 *:22 *:*
root zeek 27934 17 tcp46 *:47765 *:*
root zeek 20657 17 tcp46 *:47764 *:*
root zeek 84818 16 tcp46 *:47763 *:*
root zeek 91782 16 tcp46 *:47762 *:*
root zeek 94252 17 tcp46 *:47761 *:*
root owlhnode 334 6 tcp46 *:50002 *:*
root nfsd 46617 5 tcp4 *:2049 *:*
root mountd 37746 8 udp4 *:650 *:*
root mountd 37746 9 tcp4 *:650 *:*
root rpcbind 52182 9 udp4 *:111 *:*
root rpcbind 52182 10 udp4 *:947 *:*
root rpcbind 52182 11 tcp4 *:111 *:*
? ? ? ? udp4 *:2049 *:*

- nc command (also ipv6 works):

root@fbsdzeek01:~ # nc -zv 127.0.0.1 47761
Connection to 127.0.0.1 47761 port [tcp/*] succeeded!

- broker.log:

{"ts":"2020-02-19T08:13:35.215215Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10007,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:35.214435Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:37.198510Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10008,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:37.198165Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10009,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:36.965614Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.269695Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10010,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.275816Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10011,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:36.965614Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47762,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.271616Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10012,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.505503Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"::ffff:127.0.0.1","peer.bound_port":10013,"message":"handshake successful"}
{"ts":"2020-02-19T08:13:41.579196Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}
{"ts":"2020-02-19T08:13:41.964270Z","ty":"Broker::STATUS","ev":"peer-added","peer.address":"127.0.0.1","peer.bound_port":47761,"message":"received handshake from remote core"}

As you can see, nothing strange here ... As you said, I have changed the definition of "localhost" in the node.cfg file to IP 127.0.0.1 ... and it works!

Problem solved. Many thanks Jon...

Regards,
C. L. Martinez