Getting a Broctl Stack Trace

Zeek
1

I’m having an issue with broctl crashing when I try to run it on Alpine Linux. I mentioned it previously [1] but I’m circling back around to try to get it resolved. I’ve built it with the appropriate patches [2] but broctl is still reporting “crashed” state when I checks the status after starting it. The bro binary itself runs fine.

What do I need to do to collect a stack trace from broctl to determine the root cause?

Bro is built in debug mode and I set “ulimit -c unlimited” per the instructions on reporting problems. I see a /usr/local/bro/spool/tmp/post-terminate-standalone-2018-11-02-02-56-06-13765-crash directory but there’s no core dump anywhere obvious. The .crash-diag.out file says “No core file found” and doesn’t provide any useful information about the cause of the crash.

Thanks,
Mike

[1] http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013580.html

[2] http://mailman.icsi.berkeley.edu/pipermail/bro/2018-September/013581.html

2

Make sure you are setting the core pattern on your system so that the core dump will be written into the CWD.

sudo sysctl -w kernel.core_pattern="core.%e-%t-%p"

   .Seth

3

I gave this a shot but I’m still not seeing a core file. I tried both the setting you recommended and setting an absolute path to /tmp. When I force a core dump on another process the core file shows up as expected, but broctl isn’t producing one.

I’m open to suggestions on this one… not sure how to determine the root cause.

thanks,
Mike

4

I’m not 100% sure about the root cause, but I know one thing that may help… there’s a code path for ‘broctl start’ that will say something has “crashed” when it is “not running”… but “not running” doesn’t have to be a segfault… just that it didn’t fully initialize in the way that broctl was expecting it to.

Hosom was also looking into this today and said he was seeing:

warning in /usr/local/bro/share/bro/base/init-bare.bro, line 1: problem initializing NB-DNS: no valid nameservers in resolver config

It may be a red herring, but that’s easy to rule out:

add

env_vars=BRO_DNS_FAKE=1

to broctl.cfg

and deploy and see if things start properly with real DNS disabled.

5

Thank you for the suggestion. Adding that setting to broctl.cfg didn’t make a difference, but I did notice that when I run broctl deploy I get “(bro still initializing)” at the end. Since I don’t normally see that it makes me think something isn’t coming up correctly, rather than an actual segfault.

Are there other options I should try setting in broctl.cfg, or anything else I can do to diagnose what’s not working as broctl expects?

thanks,
Mike