I am trying to filter out certain IP’s from the conn.log. I added redef restrict_filters += {[“not-host”] = “not host 192.168.1.1”}; but if I do a grep on the conn.log I still see that IP showing up.
If I print the filters using zeekctl, I see it in there. Any thoughts?
Hello Lyle,
There are many ways to do this - depending on where you want to filter.
If I recall right, restrict_filters pretty much sets BPF filter before zeek
processes the packet - this might be useful in events where some IPs are over
burdening your zeek instance.
I still see that IP showing up.
Also, make sure your traffic isn't vlan tagged.
One approach I've used is this little filtering script:
module LOG_FILTERS;
export {
global FILTER_FILE = "/zeek-config/LOG_FILTERS" &redef ;
redef Config::config_files += { FILTER_FILE };
option filter_ips: set[addr] = set() ;
}
hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
{
if ( rec$id$resp_h in filter_ips )
break;
}
Here, I am filtering based on response IP but you can choose any of the rec$id
fields (orig_h, resp_h, orig_p, resp_p )
Advantages:
1) This uses log_policy hooks (so somewhat future proof)
2) It still lets Zeek process the data but not log in the logs
3) I've used configuration framework so basically I don't need to restart zeek
time & again to add new IPs. I can just add to config file and adds(or removes)
get propagated automatically (its cluster safe too)
Hope this helps.
Aashish
I just want to make sure that I understand the script:
"/zeek-config/LOG_FILTERS" is a file location with LOG_FILTERS as the file that contains the IP's?
If I am correct on the above, do the IP's need to be in any special format such as (not 192.168.1.1) or can it just be the IP's?
Will this also work with networks that I want to exclude, like O365 network ranges?
Ah! Sorry, should have clearified that this construct uses configuration framework:
https://docs.zeek.org/en/v3.0.14/frameworks/configuration.html
LOG_FILTERS is a file with format of
filter_ips 192.168.1.1,192.168.1.2
If you want to filter network ranges thats a subnet type so you'd need
something on the lines of script below:
Here is how file will look like:
$ cat LOG_FILTERS
filter_ips 192.168.1.1,192.168.1.2
filter_subnets 10.0.0.0/24, 10.0.1.0/24, 10.0.2.0/24
Here will how script will look like:
$ cat filter-logs.zeek
module LOG_FILTERS;
export {
global FILTER_FILE = "/zeek-config/LOG_FILTERS" &redef ;
redef Config::config_files += { FILTER_FILE };
option filter_ips: set[addr] = set() ;
option filter_subnets: set[subnet] = set() ;
}
hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter) {
if ( rec$id$resp_h in filter_ips )
break;
# change to what you want to filter out - src or dst
if (rec$id$orig_h in filter_subnets)
break;
}
Also if you are going the restrict_filter route - check for vlan tagged traffic
redef restrict_filters += {["not-host"] = "vlan and not host 192.168.1.1"};
Aashish
Thank you so much for explaining! It was very helpful.
I wish that zeek had a simplier way to exclude IP's/ports/networks
Doesn’t a plain BPF filter provide that simple case of ignoring certain traffic?
Darren,
Where would the BPF filter be added?
I initially tried adding this to the local.zeek file:
redef restrict_filters += {[“not-net”] = “not net 13.107.6.152/31 and not net 13.107.18.10/31 and not net 13.107.128.0/22 and not net 23.103.160.0/20 and not net 40.96.0.0/13 and not net 40.104.0.0/15 and not net 52.96.0.0/14 and not net 131.253.33.215/32 and not net 132.245.0.0/16 and not net 150.171.32.0/22 and not net 204.79.197.215/32 and not net 192.169.0.0/16 and not net 147.0.219.0/24”};
When I tailed the conn.log (tail -f -n 200 ./json_streaming_conn.log | grep -i 192.168) I was still seeing files from things on the 192.168.0.0 subnet.
If I add vlan to the filter, for whatever reason, zeek goes to 0% CPU and not all of the logs show up in the current log directory…almost like it’s not even running.
This message was in the notice.log:
{"_path":“notice”,"_write_ts":“2021-08-12T22:06:18.986074Z”,“ts”:“2021-08-12T22:06:18.986074Z”,“note”:“CaptureLoss::Too_Little_Traffic”,“msg”:“Only observed 0 TCP ACKs and was expecting at least 1.”,“peer_descr”:“worker-1-2”,“actions”:[“Notice::ACTION_LOG”],“suppress_for”:3600.0}
redef restrict_filters += {[“not-net”] = “vlan not net 13.107.6.152/31 and not net 13.107.18.10/31 and not net 13.107.128.0/22 and not net 23.103.160.0/20 and not net 40.96.0.0/13 and not net 40.104.0.0/15 and not net 52.96.0.0/14 and not net 131.253.33.215/32 and not net 132.245.0.0/16 and not net 150.171.32.0/22 and not net 204.79.197.215/32 and not net 10.50.0.0/16 and not net 10.51.0.0/16 and not net 147.0.219.0/24”};
Aashish,
When I copied your script and again do a tail on the conn.log with the LOG_FILTERS file in
“/opt/bro/etc/LOG_FILTERS” containing the IP 192.168.1.1, I still see the logs. I’m at a total loss. This is rather frustrating!
Example:
{"_path":“conn”,"_write_ts":“2021-08-12T21:51:49.553378Z”,“ts”:“2021-08-12T21:51:39.347321Z”,“uid”:“CnTyrN1N0x4nll4vG7”,“id.orig_h”:“xx.xx.xx.xx”,“id.orig_p”:38913,“id.resp_h”:“192.168.1.1”,“id.resp_p”:53,“proto”:“udp”,“service”:“dns”,“duration”:0.00020503997802734375,“orig_bytes”:61,“resp_bytes”:125,“conn_state”:“SF”,“local_orig”:true,“local_resp”:true,“missed_bytes”:0,“history”:“Dd”,“orig_pkts”:1,“orig_ip_bytes”:89,“resp_pkts”:1,“resp_ip_bytes”:153}
Just to make sure I didn’t screw something up, here is the script:
module LOG_FILTERS;
export {
global FILTER_FILE = “/opt/bro/etc/LOG_FILTERS” &redef ;
redef Config::config_files += { FILTER_FILE };
option filter_ips: set[addr] = set() ;
option filter_subnets: set[subnet] = set() ;
}
hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter) {
if ( rec$id$resp_h in filter_ips )
break;
if (rec$id$orig_h in filter_subnets)
break;
}
I am sorry, its not this complicated but I wish I knew the art of simplicity! 
If I add vlan to the filter, for whatever reason, zeek goes to 0% CPU and not all of the logs show up in the current log directory…almost like it’s not even running.
redef restrict_filters += {["not-net"] = "vlan not net 13.107.6.152/31 and not net 13.107.18.10/31 and not net 13.107.128.0/22 and not net 23.103.160.0/20 and not net 40.96.0.0/13 and not net 40.104.0.0/15 and not net 52.96.0.0/14 and not net 131.253.33.215/32 and not net 132.245.0.0/16 and not net 150.171.32.0/22 and not net 204.79.197.215/32 and not net 10.50.0.0/16 and not net 10.51.0.0/16 and not net 147.0.219.0/24"};
This is wrong filter.
You probably need: vlan and (not net 13.107.6.152/31 ...) instead of "vlan
not net ..."
Likewise for other IP based filter.
As I mentioned before restrict_filters is a heavy sword. First determine what
you want to do - (i) do you want zeek to not process any of this traffic or (ii) do you
just want entries specific to these IPs/subnets not in the logs, specifically
conn.log
for (i) use restrict_filter approach!
for (ii) use the script I gave you - Please pay attention to if conditions in
there!
Example snippet I gave you only filtered response IPs
if ( rec$id$resp_h in filter_ips )
break;
Likewise filter only if src IP is in the subnets
# change to what you want to filter out - src or dst
if (rec$id$orig_h in filter_subnets)
So yeah, you need to modify the if condition to your needs.
Here I modified one for you, assuming that you want to filter based on both src
and destination.
$cat LOG_FILTERS
LOG_FILTERS::filter_ips 192.168.0.1,192.168.0.2
LOG_FILTERS::filter_subnets 13.107.6.152/31, 13.107.18.10/31, 13.107.128.0/22,
zeek script:
module LOG_FILTERS;
export {
global FILTER_FILE = "LOG_FILTERS" &redef ;
redef Config::config_files += { FILTER_FILE };
option filter_ips: set[addr] = set() ;
option filter_subnets: set[subnet] = set() ;
}
hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter) {
# filter out if src or dst is in filter_ips set
if ( rec$id$resp_h in filter_ips || rec$id$orig_h in filter_ips)
break;
# filter out if src or dst is in filter_subnets set
# change to what you want to filter out - src or dst
if (rec$id$resp_h in filter_subnets || rec$id$orig_h in filter_subnets)
break;
}
Please don’t get me wrong, I appreciate the guidance…you’ve been a huge help!
Darren,
Where would the BPF filter be added?I initially tried adding this to the local.zeek file:
Seems like this thread generally documents working methods some more complex than others, so nothing really new here but confirmation that it generally works.
I’ve done this a couple of ways, referring to https://docs.zeek.org/en/master/scripts/base/frameworks/packet-filter/main.zeek.html.
In local.zeek:
redef PacketFilter::default_capture_filter = “not host 10.0.15.2”;
Or:
redef restrict_filters += {
[“unmonitored host”] = “not host 10.0.15.2”
};
Testing:
$ tcpdump -nr test.pcap
reading from file tmp.pcap, link-type EN10MB (Ethernet)
19:32:11.878769 IP 10.0.1.4 > 10.0.15.1: ICMP echo request, id 4750, seq 1, length 64
19:32:24.590370 IP 10.0.1.4 > 10.0.15.2: ICMP echo request, id 4754, seq 1, length 64
19:32:36.334450 IP 10.0.1.4 > 10.0.15.3: ICMP echo request, id 4806, seq 1, length 64
^ The PCAP has traffic in it to 3 dest hosts, including the one I exclude in the Zeek config.
zeek -C -r ../test.pcap local
zeek-cut < conn.log
1628994731.878769 CLcN4g8hTL0WsMIr9 10.0.1.4 8 10.0.15.1 0 icmp - - -
^ Zeek outputs connection records for 2 hosts and ignores the third that is filtered out by the PCAP.
Verify the effective BPF filter in packet_filter.log, and also possible to run the filter against live traffic or PCAP with tcpdump(8) to test assumptions.
1628995458.938947 zeek (ip or not ip) and (not host 10.0.15.2) T T
Is this something lacking from the zeek documentation that we could be more clear about?
Tim