Zeek BPF Limitations

Zeek
1

Hi All,

We have a need to bypass a large number of subnets from capture on a centralised Zeek sensor. The reason being, we already have an estate of remote sensors capturing for said subnets (lots of remote sensors to capture MPLS any-to-any traffic). On the centralised sensor, we appear to be hitting some limitation on the number of entries we can use.

For example -

ZeekArgs = -f "ip or not ip and not (net 10.21.37.0/24 or net 10.21.38.0/24 or net 10.21.128.0/22 or net 10.21.176.0/22 or net 10.21.16.0/24 or net 10.21.76.0/24 or net 10.21.186.0/24 or net 10.21.112.0/23 or net 10.21.12.0/23 or net 10.21.48.0/23 or net 10.21.120.0/24 or net 10.21.64.0/23 or net 10.21.235.0/24 or net 10.21.202.0/24 or net 10.21.154.0/24 or net 10.21.210.0/24 or net 10.21.185.0/24 or net 10.21.108.0/24 or net 10.21.155.0/24 or net 10.21.162.0/24 or net 10.21.184.0/24 or net 10.21.182.0/24 or net 10.21.196.0/24 or net 10.21.153.0/24 or net 10.21.222.0/24 or net 10.21.238.0/24 or net 10.21.224.0/24 or net 10.21.163.0/24 or net 10.21.187.0/24 or net 10.21.234.0/24 or net 10.21.191.0/24 or net 10.21.205.0/24 or net 10.21.31.0/24 or net 10.21.40.0/24 or net 10.21.78.0/23 or net 172.16.8.0/21 or host 10.21.91.13 or host 10.21.100.125 or host 10.21.100.164 or host 10.21.100.131 or host 10.21.100.12 or host 10.21.91.254 or host 172.26.1.24 or host 10.25.26.221)"

The above will work, but as soon as we add one more entry, /var/log/bro/current shows no logs (other than broker, cluster, stdout, etc). Remove the last entry, /var/log/bro/current fills up with expected logs again.

We’re using Debian 10 for this sensor -

  Static hostname: sensor00
         Icon name: computer-desktop
           Chassis: desktop
        Machine ID: 6347d74af08c453d82787d516fd1bdb7
           Boot ID: 47fd92ad927640e3928f756a76d8fcef
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-18-amd64
      Architecture: x86-64

Zeek version is 4.0.5.

Does anyone have any ideas why we might be hitting what looks like a BPF limitation (perhaps something to do with the underlying capture library?).

We’re running the following node.cfg -

[logger-1]
type=logger
host=localhost
#
[manager]
type=manager
host=localhost
#
[proxy-1]
type=proxy
host=localhost
#
[worker-1]
type=worker
host=localhost
interface=enp2s0
lb_method=pf_ring
lb_procs=4

I’d be very grateful if anyone has any ideas as to why we’re seeing this issue, and can offer any advice.

Cheers
Andy

2

Hello,

Are you able to capture or omit what you expect if you pass that filter to Tcpdump?

Sincerely,

Richard

3

Hi Richard,

Yes, I am, even with the full list of subnets we want to ignore.

tcpdump -i enp2s0 -w /tmp/test.pcap not net 10.22.204.0/24 or net 10.22.208.0/24 or net 10.22.135.0/24 or net 10.22.128.0/24 or net 10.22.203.0/24 or net 10.22.172.0/24 or net 10.22.167.0/24 or net 10.22.181.0/24 or net 10.22.190.0/24 or net 10.22.140.0/24 or net 10.22.173.0/24 or net 10.22.136.0/24 or net 10.22.179.0/24 or net 10.22.200.0/24 or net 10.22.198.0/24 or net 10.22.199.0/24 or net 10.22.202.0/24 or net 10.22.137.0/24 or net 10.22.165.0/24 or net 10.22.169.0/24 or net 10.22.176.0/24 or net 10.22.182.0/24 or net 10.21.37.0/24 or net 10.21.38.0/24 or net 10.21.128.0/22 or net 10.21.176.0/22 or net 10.21.16.0/24 or net 10.21.76.0/24 or net 10.21.186.0/24 or net 10.21.112.0/23 or net 10.21.12.0/23 or net 10.21.48.0/23 or net 10.21.120.0/24 or net 10.21.64.0/23 or net 10.21.235.0/24 or net 10.21.202.0/24 or net 10.21.154.0/24 or net 10.21.210.0/24 or net 10.21.185.0/24 or net 10.21.108.0/24 or net 10.21.155.0/24 or net 10.21.162.0/24 or net 10.21.184.0/24 or net 10.21.182.0/24 or net 10.21.196.0/24 or net 10.21.153.0/24 or net 10.21.222.0/24 or net 10.21.238.0/24 or net 10.21.224.0/24 or net 10.21.163.0/24 or net 10.21.187.0/24 or net 10.21.234.0/24 or net 10.21.191.0/24 or net 10.21.205.0/24 or net 10.21.31.0/24 or net 10.21.40.0/24 or net 10.21.78.0/23 or net 10.22.182.0/24 or net 172.16.8.0/21 or host 10.21.91.13 or host 10.21.100.125 or host 10.21.100.164 or host 10.21.100.131 or host 10.21.100.12 or host 10.21.91.254 or host 172.26.1.24 or host 10.25.26.221
tcpdump: listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C388542 packets captured
388542 packets received by filter
0 packets dropped by kernel

Best regards
Andy

4

Thanks, that is great – it helps rule out issues with the shell or other non-Zeek factors.

Sincerely,

Richard

5

I actually came to post about this today and was surprised someone else recently just ran into this too. My difference is I’m trying to filter approximately 3000 individual IP addresses. Seeing the same behavior as @millap with getting a broker.log, cluster.log, stderr.log and stdout.log in my Zeek logs current directory.

I believe that happens because my logger, manager and proxy come up whereas my workers are stuck initializing. I’ve left it sit all day on a lab system and they never start.

zeekctl status
waiting for lock (owned by PID 8558) ...
Name         Type    Host             Status    Pid    Started
logger       logger  127.0.0.1        running   7189   29 Sep 14:55:05
manager      manager 127.0.0.1        running   7247   29 Sep 14:55:06
proxy        proxy   127.0.0.1        running   7305   29 Sep 14:55:08
worker-1-1   worker  127.0.0.1        initializing 7454   29 Sep 14:55:09
worker-1-2   worker  127.0.0.1        initializing 7447   29 Sep 14:55:09
worker-1-3   worker  127.0.0.1        initializing 7451   29 Sep 14:55:09
worker-1-4   worker  127.0.0.1        initializing 7450   29 Sep 14:55:09
worker-1-5   worker  127.0.0.1        initializing 7458   29 Sep 14:55:09
worker-1-6   worker  127.0.0.1        initializing 7457   29 Sep 14:55:09
worker-1-7   worker  127.0.0.1        initializing 7453   29 Sep 14:55:09
worker-1-8   worker  127.0.0.1        initializing 7459   29 Sep 14:55:09
6

Hiya,

In my case, my process starts just fine -

root@sensor00:/opt/zeek/etc# zeekctl deploy
checking configurations ...
installing ...
removing old policies in /opt/zeek/spool/installed-scripts-do-not-touch/site ...
removing old policies in /opt/zeek/spool/installed-scripts-do-not-touch/auto ...
creating policy directories ...
installing site policies ...
generating cluster-layout.zeek ...
generating local-networks.zeek ...
generating zeekctl-config.zeek ...
generating zeekctl-config.sh ...
stopping ...
stopping workers ...
stopping proxy ...
stopping manager ...
stopping logger ...
starting ...
starting logger ...
starting manager ...
starting proxy ...
starting workers ...
root@sensor00:/opt/zeek/etc# zeekctl status
Name         Type    Host             Status    Pid    Started
logger-1     logger  localhost        running   9502   29 Sep 22:51:24
manager      manager localhost        running   9564   29 Sep 22:51:25
proxy-1      proxy   localhost        running   9619   29 Sep 22:51:27
worker-1-1   worker  localhost        running   9704   29 Sep 22:51:28
worker-1-2   worker  localhost        running   9714   29 Sep 22:51:28
worker-1-3   worker  localhost        running   9716   29 Sep 22:51:28
worker-1-4   worker  localhost        running   9715   29 Sep 22:51:28

I can even see the filter in the zeekctl diag output -

==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p logger-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto -f ip or not ip and not (net 10.21.37.0/24 or net 10.21.38.0/24 or net 10.21.128.0/22 or net 10.21.176.0/22 or net 10.21.16.0/24 or net 10.21.76.0/24 or net 10.21.186.0/24 or net 10.21.112.0/23 or net 10.21.12.0/23 or net 10.21.48.0/23 or net 10.21.120.0/24 or net 10.21.64.0/23 or net 10.21.235.0/24 or net 10.21.202.0/24 or net 10.21.154.0/24 or net 10.21.210.0/24 or net 10.21.185.0/24 or net 10.21.108.0/24 or net 10.21.155.0/24 or net 10.21.162.0/24 or net 10.21.184.0/24 or net 10.21.182.0/24 or net 10.21.196.0/24 or net 10.21.153.0/24 or net 10.21.222.0/24 or net 10.21.238.0/24 or net 10.21.224.0/24 or net 10.21.163.0/24 or net 10.21.187.0/24 or net 10.21.234.0/24 or net 10.21.191.0/24 or net 10.21.205.0/24 or net 10.21.31.0/24 or net 10.21.40.0/24 or net 10.21.78.0/23 or net 10.237.182.0/24 or net 172.16.8.0/21 or host 10.21.91.13 or host 10.21.100.125 or host 10.21.100.164 or host 10.21.100.131 or host 10.21.100.12 or host 10.21.91.254 or host 172.26.1.24 or host 10.25.26.221)

But no traffic is captured.

On yours, you appear to have a process lock which may be stopping your workers from completing init.

Andy

7

I think you’re on to something here. I just did a quick experiment with
a script that creates progressively larger filters and the processing
time of a small pcap deteriorates horribly.

The filter just “or”'s a given number of hosts:

$ ./megafilter.py 3
host 10.0.0.2 or host 10.0.0.3 or host 10.0.0.4

These are processing times I see for larger filters (but not so large
that the shell would balk at the command):

$ for i in 100 200 300 400 500 600; do echo; echo $i; time zeek -r
~/proj/traces/http-single.trace -f “$(./megafilter.py $i)”; done

100

real 0m0.554s
user 0m0.500s
sys 0m0.044s

200

real 0m0.844s
user 0m0.776s
sys 0m0.054s

300

real 0m1.865s
user 0m1.796s
sys 0m0.056s

400

real 0m4.880s
user 0m4.805s
sys 0m0.062s

500

real 0m11.498s
user 0m11.235s
sys 0m0.111s

600

real 0m22.481s
user 0m22.168s
sys 0m0.196s

gdb says these things are stuck in pcap_compile(), which fits.

Not sure what we can do here but I’ll create a ticket so we can look
into it further.

Best,
Christian

8

@millap, your case seems different from just slow filter compilation time … do you have any clues in packet_filter.log, notice.log, zeekctl capstats, zeekctl netstats, etc? Could it be that by accident the additional filter modification just excludes all traffic?

Best,
Christian

9

Thanks for looking into it! I meant to reply to @millap about the lock. That does eventually go away but my workers remain stuck initializing for forever.

Is the BPF the only means of preventing Zeek from analyzing packets from specific IPs and subnets?

Unfortunately, my Zeek boxes are on a tap that sees our vulnerability scanning, and it causes severe memory problems. While not all 3,000 IPs I’m looking to block come across the same tap, it’s been a challenging task to find enough network documentation to know which IP is going to go where.

Thanks!

10

Is the BPF the only means of preventing Zeek from analyzing packets from specific IPs and subnets?

In general, yes. Some sites also push blocklists dynamically to their taps’ switches/routers, via Zeek’s NetControl framework. The docs have some pointers on this. Are those 3K addresses scattered all over the place, or could you perhaps aggregate them into subnets?

Here’s the ticket btw: Zeek deals poorly with long `pcap_compile()` times · Issue #2457 · zeek/zeek · GitHub

Best,
Christian

11

Hi Christian,

Thanks for your response. It’s very much appreciated.

Running a deploy with the extended filter shows -

root@sensor00:/opt/zeek/etc# zeekctl deploy
checking configurations ...
installing ...
removing old policies in /opt/zeek/spool/installed-scripts-do-not-touch/site ...
removing old policies in /opt/zeek/spool/installed-scripts-do-not-touch/auto ...
creating policy directories ...
installing site policies ...
generating cluster-layout.zeek ...
generating local-networks.zeek ...
generating zeekctl-config.zeek ...
generating zeekctl-config.sh ...
stopping ...
stopping workers ...
stopping proxy ...
stopping manager ...
stopping logger ...
starting ...
starting logger ...
starting manager ...
starting proxy ...
starting workers ...

The output you requested is -

root@sensor00:/var/log/bro/current# zeekctl capstats
Interface             kpps       mbps       (10s average)
----------------------------------------
localhost/enp2s0      26.2       153.9

And

root@sensor00:/var/log/bro/current# zeekctl netstats
 worker-1-1: 1665081145.737340 recvd=0 dropped=0 link=0
 worker-1-2: 1665081145.743049 recvd=0 dropped=0 link=0
 worker-1-3: 1665081145.748770 recvd=0 dropped=0 link=0
 worker-1-4: 1665081145.757964 recvd=0 dropped=0 link=0

The notice log shows -

{"ts":1665081154.163262,"note":"CaptureLoss::Too_Little_Traffic","msg":"Only observed 0 TCP ACKs and was expecting at least 1.","peer_descr":"worker-1-3","actions":["Notice::ACTION_LOG"],"suppress_for":3600.0}
{"ts":1665081154.1701,"note":"CaptureLoss::Too_Little_Traffic","msg":"Only observed 0 TCP ACKs and was expecting at least 1.","peer_descr":"worker-1-2","actions":["Notice::ACTION_LOG"],"suppress_for":3600.0}
{"ts":1665081154.187173,"note":"CaptureLoss::Too_Little_Traffic","msg":"Only observed 0 TCP ACKs and was expecting at least 1.","peer_descr":"worker-1-4","actions":["Notice::ACTION_LOG"],"suppress_for":3600.0}
{"ts":1665081154.195957,"note":"CaptureLoss::Too_Little_Traffic","msg":"Only observed 0 TCP ACKs and was expecting at least 1.","peer_descr":"worker-1-1","actions":["Notice::ACTION_LOG"],"suppress_for":3600.0}

packet_filter.log is empty.

However if I run a tcpdump using the same excluded subnets, I get -

root@sensor00:/opt/zeek/etc# tcpdump -i enp2s0 -s 0 -w /tmp/temp.pcap not net 10.22.128.0/24 or net 10.21.37.0/24 or net 10.21.38.0/24 or net 10.21.128.0/23 or net 10.21.130.0/23 or net 10.21.177.0/24 or net 10.21.178.0/24 or net 10.21.179.0/24 or net 10.21.16.0/24 or net 10.21.76.0/24 or net 10.21.186.0/24 or net 10.21.112.0/23 or net 10.21.12.0/23 or net 10.21.48.0/23 or net 10.21.120.0/24 or net 10.21.64.0/23 or net 10.21.235.0/24 or net 10.21.202.0/24 or net 10.21.154.0/24 or net 10.22.204.0/24 or net 10.22.208.0/24 or net 10.21.210.0/24 or net 10.21.185.0/24 or net 10.22.165.0/24 or net 10.22.169.0/24 or net 10.22.176.0/24 or net 10.22.182.0/24 or net 10.21.108.0/24 or net 10.21.155.0/24 or net 10.21.162.0/24 or net 10.21.184.0/24 or net 10.21.182.0/24 or net 10.21.196.0/24 or net 10.22.135.0/24 or net 10.21.153.0/24 or net 10.22.202.0/24 or net 10.21.222.0/24 or net 10.21.238.0/24 or net 10.22.198.0/24 or net 10.22.199.0/24 or net 10.21.224.0/24 or net 10.21.163.0/24 or net 10.21.187.0/24 or net 10.22.136.0/24 or net 10.21.234.0/24 or net 10.22.179.0/24 or net 10.22.200.0/24 or net 10.21.191.0/24 or net 10.22.173.0/24 or net 10.21.205.0/24 or net 10.21.176.0/24 or net 10.21.220.0/24 or net 10.22.181.0/24 or net 10.22.190.0/24 or net 10.22.140.0/24 or net 10.21.86.0/24 or net 10.21.152.0/24 or net 10.21.223.0/24 or net 10.21.31.0/24 or net 10.21.40.0/24 or net 10.21.237.0/24 or net 10.22.172.0/24 or net 10.22.167.0/24 or net 10.21.156.0/24 or net 10.21.195.0/24 or net 10.22.137.0/24 or net 10.21.21.0/24 or net 10.21.150.0/24 or net 10.21.199.0/24 or net 10.21.151.0/24 or net 10.21.168.0/24 or net 10.21.183.0/24 or net 10.21.192.0/24 or net 10.21.165.0/24 or net 10.21.78.0/23 or net 10.22.203.0/24 or net 172.16.8.0/21 or host 10.21.91.13 or host 10.21.100.125 or host 10.21.100.164 or host 10.21.100.131 or host 10.21.100.12 or host 10.21.91.254 or host 172.26.1.24 or host 10.25.26.221
tcpdump: listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C335414 packets captured
335414 packets received by filter
0 packets dropped by kernel

The temp.pcap file is this size after a few seconds -

-rw-r--r-- 1 root root 252M Oct 6 19:35 temp.pcap

The data inside looks complete based on my experience of the customer environment.

Let me know if I can grab anything else to help determine the problem.

Best regards
Andy

12

Thanks Andy. It all seems to confirm your observation. Are you able to skip zeekctl and try some Zeek invocations directly, i.e. something like zeek -i enp2s0 -f '...'? It’d be good to see confirmation of the problem with the exact filters you’re setting.

Best,
Christian

13

Another thought: try a zeekctl setup with the same offending filter but without PF_RING, and see if it makes a difference.

Best,
Christian

14

Hi Christian,

Running Zeek manually -

root@sensor00:/opt/zeek/etc# zeek -i enp2s0 -f 'ip or not ip and not (net 10.22.128.0/24 or net 10.21.37.0/24 or net 10.21.38.0/24 or net 10.21.128.0/23 or net 10.21.130.0/23 or net 10.21.177.0/24 or net 10.21.178.0/24 or net 10.21.179.0/24 or net 10.21.16.0/24 or net 10.21.76.0/24 or net 10.21.186.0/24 or net 10.21.112.0/23 or net 10.21.12.0/23 or net 10.21.48.0/23 or net 10.21.120.0/24 or net 10.21.64.0/23 or net 10.21.235.0/24 or net 10.21.202.0/24 or net 10.21.154.0/24 or net 10.22.204.0/24 or net 10.22.208.0/24 or net 10.21.210.0/24 or net 10.21.185.0/24 or net 10.22.165.0/24 or net 10.22.169.0/24 or net 10.22.176.0/24 or net 10.22.182.0/24 or net 10.21.108.0/24 or net 10.21.155.0/24 or net 10.21.162.0/24 or net 10.21.184.0/24 or net 10.21.182.0/24 or net 10.21.196.0/24 or net 10.22.135.0/24 or net 10.21.153.0/24 or net 10.22.202.0/24 or net 10.21.222.0/24 or net 10.21.238.0/24 or net 10.22.198.0/24 or net 10.22.199.0/24 or net 10.21.224.0/24 or net 10.21.163.0/24 or net 10.21.187.0/24 or net 10.22.136.0/24 or net 10.21.234.0/24 or net 10.22.179.0/24 or net 10.22.200.0/24 or net 10.21.191.0/24 or net 10.22.173.0/24 or net 10.21.205.0/24 or net 10.21.176.0/24 or net 10.21.220.0/24 or net 10.22.181.0/24 or net 10.22.190.0/24 or net 10.22.140.0/24 or net 10.21.86.0/24 or net 10.21.152.0/24 or net 10.21.223.0/24 or net 10.21.31.0/24 or net 10.21.40.0/24 or net 10.21.237.0/24 or net 10.22.172.0/24 or net 10.22.167.0/24 or net 10.21.156.0/24 or net 10.21.195.0/24 or net 10.22.137.0/24 or net 10.21.21.0/24 or net 10.21.150.0/24 or net 10.21.199.0/24 or net 10.21.151.0/24 or net 10.21.168.0/24 or net 10.21.183.0/24 or net 10.21.192.0/24 or net 10.21.165.0/24 or net 10.21.78.0/23 or net 10.22.203.0/24 or net 172.16.8.0/21 or host 10.21.91.13 or host 10.21.100.125 or host 10.21.100.164 or host 10.21.100.131 or host 10.21.100.12 or host 10.21.91.254 or host 172.26.1.24 or host 10.25.26.221)'
listening on enp2s0

Warning: Kernel filter failed: Cannot allocate memory
0 packets received on interface not open, 0 (0.00%) dropped

Best regards
Andy

15

Hi Christian,

Running zeekctl as requested -

==== .status
RUNNING [run_loop]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[worker-1]

No core file found and gdb is not installed.  It is recommended to
install gdb so that ZeekControl can output a backtrace if Zeek crashes.

Zeek 4.0.5
Linux 4.19.0-21-amd64

Zeek plugins: (none found)

==== No reporter.log

==== stderr.log
warning in /opt/zeek/spool/installed-scripts-do-not-touch/site/local.zeek, line 132: deprecated (Log::Filter$pred): Remove in 4.1. PolicyHooks will replace the $pred function.
listening on enp2s0

Warning: Kernel filter failed: Cannot allocate memory

==== stdout.log
max memory size         (kbytes, -m) unlimited
data seg size           (kbytes, -d) unlimited
virtual memory          (kbytes, -v) unlimited
core file size          (blocks, -c) unlimited

==== .cmdline
-i enp2s0 -U .status -p zeekctl -p zeekctl-live -p local -p worker-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto -f ip or not ip and not (net 10.22.128.0/24 or net 10.21.37.0/24 or net 10.21.38.0/24 or net 10.21.128.0/23 or net 10.21.130.0/23 or net 10.21.177.0/24 or net 10.21.178.0/24 or net 10.21.179.0/24 or net 10.21.16.0/24 or net 10.21.76.0/24 or net 10.21.186.0/24 or net 10.21.112.0/23 or net 10.21.12.0/23 or net 10.21.48.0/23 or net 10.21.120.0/24 or net 10.21.64.0/23 or net 10.21.235.0/24 or net 10.21.202.0/24 or net 10.21.154.0/24 or net 10.22.204.0/24 or net 10.22.208.0/24 or net 10.21.210.0/24 or net 10.21.185.0/24 or net 10.22.165.0/24 or net 10.22.169.0/24 or net 10.22.176.0/24 or net 10.22.182.0/24 or net 10.21.108.0/24 or net 10.21.155.0/24 or net 10.21.162.0/24 or net 10.21.184.0/24 or net 10.21.182.0/24 or net 10.21.196.0/24 or net 10.22.135.0/24 or net 10.21.153.0/24 or net 10.22.202.0/24 or net 10.21.222.0/24 or net 10.21.238.0/24 or net 10.22.198.0/24 or net 10.22.199.0/24 or net 10.21.224.0/24 or net 10.21.163.0/24 or net 10.21.187.0/24 or net 10.22.136.0/24 or net 10.21.234.0/24 or net 10.22.179.0/24 or net 10.22.200.0/24 or net 10.21.191.0/24 or net 10.22.173.0/24 or net 10.21.205.0/24 or net 10.21.176.0/24 or net 10.21.220.0/24 or net 10.22.181.0/24 or net 10.22.190.0/24 or net 10.22.140.0/24 or net 10.21.86.0/24 or net 10.21.152.0/24 or net 10.21.223.0/24 or net 10.21.31.0/24 or net 10.21.40.0/24 or net 10.21.237.0/24 or net 10.22.172.0/24 or net 10.22.167.0/24 or net 10.21.156.0/24 or net 10.21.195.0/24 or net 10.22.137.0/24 or net 10.21.21.0/24 or net 10.21.150.0/24 or net 10.21.199.0/24 or net 10.21.151.0/24 or net 10.21.168.0/24 or net 10.21.183.0/24 or net 10.21.192.0/24 or net 10.21.165.0/24 or net 10.21.78.0/23 or net 10.22.203.0/24 or net 172.16.8.0/21 or host 10.21.91.13 or host 10.21.100.125 or host 10.21.100.164 or host 10.21.100.131 or host 10.21.100.12 or host 10.21.91.254 or host 172.26.1.24 or host 10.25.26.221)

==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/zeek:/opt/zeek/bin:/usr/sbin:/sbin:/usr/bin:/bin
ZEEKPATH=/opt/zeek/spool/installed-scripts-do-not-touch/site::/opt/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site
CLUSTER_NODE=worker-1

==== .status
RUNNING [run_loop]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

Best regards
Andy

16

Warning: Kernel filter failed: Cannot allocate memory

Progress! That’s a PF_RING warning, in a code path that treats this as non-fatal. I’m not sure how/whether it gets signaled to Zeek, since it makes no sense for Zeek to continue. You’re using a PF_RING-enabled libpcap, right?

Is memory very low on that box? Seems odd that you can reliably trigger this with one extra filter condition.

Best,
Christian

17

Hi Christian,

Here’s the meminfo output -

root@sensor00:/opt/zeek/etc# cat /proc/meminfo 
MemTotal:       16319040 kB
MemFree:         1390740 kB
MemAvailable:   13273432 kB
Buffers:         1577892 kB
Cached:         10258932 kB
SwapCached:            0 kB
Active:          3537276 kB
Inactive:       10215060 kB
Active(anon):    1963980 kB
Inactive(anon):   108020 kB
Active(file):    1573296 kB
Inactive(file): 10107040 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:             17912 kB
Writeback:             0 kB
AnonPages:       1910556 kB
Mapped:           464460 kB
Shmem:            156348 kB
Slab:             614692 kB
SReclaimable:     527508 kB
SUnreclaim:        87184 kB
KernelStack:        4016 kB
PageTables:         6924 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8159520 kB
Committed_AS:    2982484 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:             3360 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1693696 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      324084 kB
DirectMap2M:    12150784 kB
DirectMap1G:     4194304 kB

We are using PF_RING, yes.

Best regards
Andy

18

Andy I’m not sure I can help further since I doubt this has anything to do with Zeek. If you’re building Zeek yourself, you could see whether the following makes things better or worse:

diff --git i/src/iosource/BPF_Program.h w/src/iosource/BPF_Program.h
index 220e67b4f..5483fb11e 100644
--- i/src/iosource/BPF_Program.h
+++ w/src/iosource/BPF_Program.h
@@ -28,13 +28,13 @@ public:
        // Parameters are like in pcap_compile(). Returns true
        // for successful compilation, false otherwise.
        bool Compile(pcap_t* pcap, const char* filter, uint32_t netmask, std::string& errbuf,
-                    bool optimize = true);
+                    bool optimize = false);

        // Creates a BPF program when no pcap handle is around,
        // similarly to pcap_compile_nopcap(). Parameters are
        // similar. Returns true on success.
        bool Compile(int snaplen, int linktype, const char* filter, uint32_t netmask,
-                    std::string& errbuf, bool optimize = true);
+                    std::string& errbuf, bool optimize = false);

        // Returns true if this program currently contains compiled
        // code, false otherwise.

If you’re able to deploy the filter in the same setup but without PF_RING, that’d be further evidence pointing at PF_RING. I’d ask the PF_RING folks either way; they may have some troubleshooting suggestions.

Best,
Christian