BRO Logger crashing due to large DNS log files

Zeek
21

Sorry, forgot to send that, I did re-enable the conn.log.

Ron

A few things look like they are still not right.

/usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54 1534945591.937813 Reporter::WARNING Your interface is likely receiving invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
/usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54 1534891136.995923 Reporter::WARNING Your interface is likely receiving invalid TCP checksums, most likely from NIC checksum offloading. By default, packets with invalid checksums are discarded by Bro unless using the -C command-line option or toggling the 'ignore_checksums' variable. Alternatively, disable checksum offloading by the network adapter to ensure Bro analyzes the actual checksums that are transmitted.
5 duplicate messages suppressed

Ensuring all the offloading is setup correctly on your interfaces may fix a few of these. I have another plugin for broctl that can do this
automatically for you:

https://packages.bro.org/packages/view/7520ca9d-4fb7-11e8-88be-0a645a3f3086

you just need to install it and then add

interfacesetup.enabled=1

to your broctl.cfg

The next time bro restarts it will ensure a bunch of settings are set properly using ethtool.

#################################################################
# Checking if many recent connections have a SAD or had history #
#################################################################
error: 24.30%, 52 out of 214 connections are half duplex

#################################################################
# Checking if many recent connections have a SAD or had history #
#################################################################
error: 75.53%, 14289 out of 18918 connections are half duplex

These 2 show that things are not working well at all. The rest of the checks don't mean much until this is fixed, so this is the thing to focus on.

This could be caused by an internal load balancing problem, or by an upstream issue. It's pretty easy to figure out which, I just haven't worked out the
best way to have bro-doctor automate it.

What you want to do is run this script from a host that bro will see the request and response:

for x in $(seq -w 1 19); do
    echo -e 'GET / HTTP/1.1\r\nHost: www.bro.org\r\n\r\n' |
    socat - tcp-connect:www.bro.org:80,sp=300$x,reuseaddr;
    sleep 1;
done

Wait a few minutes (if packets are being dropped, it may take 5 minutes for the log entries to show up only after the connections timeout) and then run

cat conn.log |bro-cut -d ts id.orig_p id.resp_h id.resp_p history orig_pkts resp_pkts|fgrep 192.150.187.43|fgrep 300

You should see something like this:

2018-08-22T11:53:46-0500 30001 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:47-0500 30002 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:48-0500 30003 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:49-0500 30004 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:50-0500 30005 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:52-0500 30006 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:53-0500 30007 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:54-0500 30008 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:55-0500 30009 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:56-0500 30010 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:57-0500 30011 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:53:59-0500 30012 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:00-0500 30013 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:01-0500 30014 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:02-0500 30015 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:03-0500 30016 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:04-0500 30017 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:06-0500 30018 192.150.187.43 80 ShADFadf 6 4
2018-08-22T11:54:07-0500 30019 192.150.187.43 80 ShADFadf 6 4

duplicate, missing, or split entries indicate different problems.

############################################################################
# Checking what percentage of recent tcp connections are remote to remote. #
############################################################################
error: 52.85%, 52853 out of 100000 connections are remote to remote

In your networks.cfg make sure you have listed all of your 'local' address space.

22

Justin,

  I disabled checksum checking, but still get the same error in the doctor output. I'm not sure if the interfacesetup config is working, using defaults for now. I'm going to get with our network guy and review the tap configurations tomorrow and make sure there is no issues there.

Thanks Again,

Ron

## Global BroControl configuration file.

interfacesetup.enabled=1
#To change the default mtu that is configured
#interfacesetup.mtu=9000

#To change the default commands that are used to bring up the interface
#interfacesetup.up_command=/sbin/ifconfig {interface} up mtu {mtu}
#interfacesetup.flags_command=/sbin/ethtool -K {interface} gro off lro off rx off tx off gso off

## If true, don't verify checksums. Useful for running on altered trace
## files, and for saving a few cycles, but at the risk of analyzing invalid
## data. Note that the ``-C`` command-line option overrides the setting of this
## variable.
const ignore_checksums = T &redef;