/misc/capture_loss percent_lost vs /misc/stats pkts dropped and missed bytes in bro_conn

Zeek
1

Hello,

In one of our bro deployments we are logging some missed byets on bro_conn logs. This is an example of a conn log with missing bytes:

"local_resp": false,
    "tunnel_parents": [],
    "local_orig": true,
    "dst_addr": "211.115.118.190",
    "src_port": 57786,
    "dst_port": 443,
    "service": "ssl",
    "duration": 0.717725,
    "resp_pkts": 28,
    "src_addr": "10.16.0.115",
    "uid": "C7H1Jb1qJhHVg05wq8",
    "history": "ShADadfF",
    "orig_pkts": 16,
    "host": "logstash",
    "conn_state": "SF",
    "orig_bytes": 2883,
    "path": "/var/log/bro/logs/current/conn.log",
    "@timestamp": "2018-10-16T09:42:14.074Z",
    "times_created": "2018-10-16T09:42:13.357Z",
    "tags": [
      "bro",
      "bro_conn"
    ],
    "proto": "tcp",
    "@version": "1",
    "resp_ip_bytes": 23649,
    "orig_ip_bytes": 3535,
    "missed_bytes": 2920,
    "resp_bytes": 22517,
    "resp_cc": "IT"
  }

I’m running both /policy/misc/capture_loss and /policy/misc/stats scripts and this is the result:
/misc/stats:

"_source": {
    "files": 40386,
    "mem": 820,
    "active_icmp_conns": 341,
    "dns_requests": 0,
    "active_tcp_conns": 6641,
    "timers": 542182,
    "peer": "worker-1-1",
    "reassem_file_size": 1040104,
    "events_proc": 2285899,
    "active_timers": 33245,
    "host": "logstash",
    "reassem_frag_size": 10528,
    "active_files": 208,
    "icmp_conns": 877,
    "events_queued": 2285898,
    "pkts_dropped": 0,
    "pkts_proc": 10232397,
    "path": "/var/log/bro/logs/current/stats.log",
    "pkts_link": 10232664,
    "udp_conns": 21084,
    "reassem_unknown_size": 0,
    "@timestamp": "2018-10-16T09:15:32.648Z",
    "pkt_lag": 0.007681,
    "active_dns_requests": 0,
    "reassem_tcp_size": 863992,
    "tags": [
      "bro",
      "bro_stats"
    ],
    "active_udp_conns": 2207,
    "tcp_conns": 27070,
    "@version": "1",
    "bytes_recv": 6580937768
  }

/misc/capture_loss:

"_source": {
    "gaps": 92247,
    "peer": "worker-1-1",
    "path": "/var/log/bro/logs/current/capture_loss.log",
    "ts_delta": 900.000031,
    "@timestamp": "2018-10-16T09:15:32.632Z",
    "percent_lost": 2.053046,
    "tags": [
      "bro",
      "bro_stats",
      "bro_capture_loss"
    ],
    "@version": "1",
    "host": "logstash",
    "acks": 4493178
  }

By reading the documentation It looks like the switch SPAN port or the network interface is dropping packets since bro stats doesn’t register any packet drops.
I’ve checked on the switch and it doesn’t report any dropped traffic.

Is this possible that the network interface of our server is dropping? Is there a way to analyze the problem further?

2

Tell us what kind of capture method you use and we will take it from here.

3

Hi, I’m using af_packet. This is my broctl.cfg file:

LogRotationInterval = 3600
LogExpireInterval = 5day
StatsLogEnable = 1
StatsLogExpireInterval = 14
StatusCmdShowAll = 0
CrashExpireInterval = 0
SitePolicyScripts = local.bro
LogDir = /var/log/bro/logs
SpoolDir = /var/log/bro/spool
CfgDir = /opt/bro/etc
lb_custom.InterfacePrefix=af_packet::

And this is my node.cfg file:

[manager]
type=manager
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=enp2s0f1
lb_method=custom
lb_procs=2
af_packet_fanout_id=21
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=128*1024*1024

I hope this helps. Thanks for your help!

4

Excellent choice!

We created (with Peter Manev, a Suricata developer) a tuning guide that also applies to Bro.

Pay special attention to these two sections

  • life of a packet
  • packet drops (near the end)

There are a few places packets can be dropped, so it’s important to know all of them.

https://github.com/pevma/SEPTun