All,
I did a quick search, but did not see any threads on this type of subject, so forgive me if this has already been discussed. We have a new bro server being stood up that looks to be creating multiple (3) entries for every conn log. Below is a sample of what I’m speaking of. We have 4 monitoring interfaces with varying numbers of CPU cores assigned to the 4 workers they are associated with. The number of entries appears to be related to the number pf_ring workers created because I changed the nodes from 3 lb_procs each to the below node.cfg config this morning and I am now seeing 1 to 5 entries for each log entry.
Would this be an indication that there is a problem with our pf_ring setup? How might we confirm what may be causing this?
$ zcat /usr/local/logs/2018-01-01/conn.01:00:00-02:00:00.log.gz |bro-cut -d |grep ‘101.124.88.146’
2018-01-01T01:16:50-0800 CxcveSg78Iow3jix 101.124.88.146 33589 137.164.29.67 53 udp dns 0.000383 44 162 SF F T 0 Dd 72 1 190 (empty)
2018-01-01T01:16:50-0800 CbvYq642taiDEEPLwc 101.124.88.146 33589 137.164.29.67 53 udp dns 0.000383 44 162 SF F T 0 Dd 72 1 190 (empty)
2018-01-01T01:16:50-0800 CEc8UA2AyEKYDSNiw9 101.124.88.146 33589 137.164.29.67 53 udp dns 0.000383 44 162 SF F T 0 Dd 72 1 190 (empty)
2018-01-01T01:17:44-0800 Clg6KI2hLQBShRUxal 101.124.88.146 54683 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 72 0 0 (empty)
2018-01-01T01:17:44-0800 CN1WXh1ae3r9FzXA7d 101.124.88.146 54683 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 72 0 0 (empty)
2018-01-01T01:17:44-0800 C7s9Qp4LPByzC6Gm53 101.124.88.146 16779 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 82 0 0 (empty)
2018-01-01T01:17:44-0800 CHBMNSB9eOhekrAS6 101.124.88.146 54683 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 72 0 0 (empty)
2018-01-01T01:17:44-0800 Cmnrmh2ivDksWcJXLl 101.124.88.146 16779 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 82 0 0 (empty)
2018-01-01T01:17:44-0800 CTCK4qkRHhDz4jLqk 101.124.88.146 16779 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 82 0 0 (empty)
2018-01-01T01:17:45-0800 CZ90QS3RGjHqck8zvc 101.124.88.146 26774 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 82 0 0 (empty)
2018-01-01T01:17:45-0800 C96L6a1YTuUCsvbHRk 101.124.88.146 26774 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 82 0 0 (empty)
2018-01-01T01:17:45-0800 CnP2AnwrmyniFfDBe 101.124.88.146 26774 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 82 0 0 (empty)
2018-01-01T01:17:46-0800 CB5iF7f4hoqrQWiq2 101.124.88.146 25389 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 72 0 0 (empty)
2018-01-01T01:17:46-0800 CkZEAp0saM0cEaTSf 101.124.88.146 25389 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 72 0 0 (empty)
2018-01-01T01:17:46-0800 CQx4sVzjmGlPnAa51 101.124.88.146 25389 137.145.204.10 53 udp dns - - - S0 F F 0 D 1 72 0 0 (empty)
$ cat /usr/local/etc/node.cfg
Example BroControl node configuration.
You're probably not really using pf_ring. Bro-doctor was written to troubleshoot problems like this..
bro-pkg install bro-doctor
broctl doctor.bro
You're either not linked against pf_ring properly, or possible you installed bro and then pf_ring, in which case a (just fixed) bug in broctl will disable pf_ring and you need to add
pfringclusterid = 11
to your broctl.cfg
You should also look into use the native bro pf_ring plugin, which is a little harder to misconfigure.
Justin,
Thanks for the quick response. Our Systems team assures me the pf_ring is compiled correctly and provided the below output. We have some ACL’s in place that makes it difficult to load the bro-doctor pkg you mention easily, but will work towards getting that tool in place. In the meantime, is there anything about the below output that looks out of place or missing? We’ll also be setting the pfringclusterid in the broctl.cfg to see if that fixes the issue.
ldd /usr/local/bin/bro | grep pcap
libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x00007f5473516000)
strings /usr/local/lib/libpcap.so.1 | grep pfring | tail -n3
pfring_mod_open
pfring_mod_get_bound_device_address
pfring_hw_ft_remove_hw_rule
Philip
Your node.cfg is slightly complicated since you’re sniffing a number of interfaces. I wouldn’t be surprised if you’re accidentally seeing the same traffic on multiple interfaces. Add this little blurb into your local.bro…
begin
@load base/protocols/conn
export {
redef record Conn::Info += {
The name of the node where this connection was analyzed.
node: string &log &optional;
};
}
event connection_state_remove(c: connection) &priority=2
{
c$conn$node = peer_description;
}
end
That will let you see which node each of your conn log entries was being written by. Look at three of the same connections to see if they’re coming from different workers and let us know.
.Seth
Seth,
Thanks for the troubleshooting code. It looks like only one interface is getting the traffic, but all 4 cores assigned are processing the same traffic individually. I’m still working with my Systems team on the suggestion from Justin.
$ cat /usr/local/logs/current/conn.log |grep -P ‘137.164.29.72’ |grep -P ‘173.245.59.100’ |grep ‘49519’
1515000332.481539 CY3pub5wesleIxzhh 137.164.29.72 49519 173.245.59.100 53 udp dns 0.001269 54 54 SF T F 0 Dd 1 82 1 82 (empty) worker-4-2
1515000332.481539 CkBMQVujDASjTbiZb 137.164.29.72 49519 173.245.59.100 53 udp dns 0.001269 54 54 SF T F 0 Dd 1 82 1 82 (empty) worker-4-1
1515000332.481539 CMTY2G1ferRjOZtWz 137.164.29.72 49519 173.245.59.100 53 udp dns 0.001269 54 54 SF T F 0 Dd 1 82 1 82 (empty) worker-4-4
1515000332.481539 CMj0iL2677DvhMHNue 137.164.29.72 49519 173.245.59.100 53 udp dns 0.001269 54 54 SF T F 0 Dd 1 82 1 82 (empty) worker-4-3
Philip
Could you try removing all of the worker configs from node.cfg except for worker-4? I'm curious if there is something we did that is causing trouble for PF_Ring if multiple interfaces are being sniffed like that.
.Seth
Seth and Justin,
It looks to be working now. The latest change that was made was dropping
the pfring version from 7.0.0 to 6.6.0. That in combination with using
the "pfringclusterid = 11" setting in the broctl.cfg got it working
correctly. We're no longer seeing any multiple entries for the same
activity.
Thanks for all the help.
Philip