Bro + pf_ring on a rasberry pi 3

Hi all!

After successfully compiling pf_ring and enable module on a rpi 3 arm kernel :

pi@raspberrypi:~ $ modinfo pf_ring && cat /proc/net/pf_ring/info
filename: /lib/modules/4.4.34-v7+/kernel/net/pf_ring/pf_ring.ko
alias: net-pf-27
description: Packet capture acceleration and analysis
author: ntop.org
license: GPL
srcversion: 159AD63EACFCF3EFC835D09
depends:
vermagic: 4.4.34-v7 SMP mod_unload modversions ARMv7
parm: min_num_slots:Min number of ring slots (uint)
parm: perfect_rules_hash_size:Perfect rules hash size (uint)
parm: transparent_mode:(deprecated) (uint)
parm: enable_debug:Set to 1 to enable PF_RING debug tracing into the syslog (uint)
parm: enable_tx_capture:Set to 1 to capture outgoing packets (uint)
parm: enable_frag_coherence:Set to 1 to handle fragments (flow coherence) in clusters (uint)
parm: enable_ip_defrag:Set to 1 to enable IP defragmentation(only rx traffic is defragmentead) (uint)
parm: quick_mode:Set to 1 to run at full speed but with upto one socket per interface (uint)
PF_RING Version : 6.4.1 (unknown)
Total rings : 2

Standard (non ZC) Options
Ring slots : 32768
Slot version : 16
Capture TX : Yes [RX+TX]
IP Defragment : No
Socket Mode : Standard
Total plugins : 0
Cluster Fragment Queue : 0
Cluster Fragment Discard : 0

I compiled also successfully bro with pf_ring plugin. But there is a problem…Although rpi interface “sees” network traffic as it is plugged on a network mirror bridge and pf_ring compiled tcpdump output does full network packet capture :

pi@raspberrypi:~/bro-test $ ifconfig
eth0 Link encap:Ethernet HWaddr b8:27:eb:68:1a:49
inet addr:10.0.0.31 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::18a4:4736:aeb7:94b7/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5912 errors:0 dropped:0 overruns:0 frame:0
TX packets:1317 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:358436 (350.0 KiB) TX bytes:166018 (162.1 KiB)

pi@raspberrypi:~/bro-test $ sudo /opt/pfring/sbin/tcpdump host not 10.0.0.31
[PF_RING] mmap() failed: try with a smaller snaplen
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:00:43.045119 IP 10.0.0.2.37630 > 10.0.0.3.9200: Flags [.], seq 2264223995:2264225443, ack 4236626719, win 1444, options [nop,nop,TS val 3506664 ecr 3496553], length 1448
21:00:43.045498 IP 10.0.0.2.37630 > 10.0.0.3.9200: Flags [.], seq 1448:2896, ack 1, win 1444, options [nop,nop,TS val 3506664 ecr 3496553], length 1448
21:00:43.045500 IP 10.0.0.2.37630 > 10.0.0.3.9200: Flags [P.], seq 2896:4096, ack 1, win 1444, options [nop,nop,TS val 3506664 ecr 3496553], length 1200
21:00:43.045502 IP 10.0.0.2.37630 > 10.0.0.3.9200: Flags [.], seq 4096:5544, ack 1, win 1444, options [nop,nop,TS val 3506664 ecr 3496553], length 1448
21:00:43.046343 IP 10.0.0.2.37630 > 10.0.0.3.9200: Flags [.], seq 5544:6992, ack 1, win 1444, options [nop,nop,TS val 3506664 ecr 3496553], length 1448
21:00:43.046344 IP 10.0.0.2.37630 > 10.0.0.3.9200: Flags [P.], seq 6992:7028, ack 1, win 1444, options [nop,nop,TS val 3506664 ecr 3496553], length 36
21:00:43.046346 IP 10.0.0.3.9200 > 10.0.0.2.37630: Flags [.], ack 7028, win 1024, options [nop,nop,TS val 3496778 ecr 3506664], length 0
^C
7 packets captured
10 packets received by filter
3 packets dropped by kernel

When i start bro with pf_ring bro exports logs only for rpi self traffic that is to say traffic from or to 10.0.0.31 ip:

pi@raspberrypi:~/bro-test $ sudo /opt/bro/bin/bro -i pf_ring::eth0
listening on eth0

1488315827.676782 616 packets received on interface eth0, 0 dropped

pi@raspberrypi:~/bro-test $ cat conn.log
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path conn
#open 2017-02-28-21-03-39
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytestunnel_parents
#types time string addr port addr port enum string interval count count string bool bool count stringcount count count count set[string]
1488315814.841388 Cn0COF2Dl2RGFtlsak 10.8.0.2 60414 10.0.0.31 22 tcp - - - - OTH – 0 ^c 0 0 0 0 (empty)
1488315826.472327 C0a8me4cjwn36bIpZ 10.8.0.2 60414 10.0.0.31 22 tcp - - - - OTH – 0 ^c 0 0 0 0 (empty)
#close 2017-02-28-21-03-47

There are no errors and no capture_loss or drop packets, although base bro plugins are enable, bro sees only limited events:

pi@raspberrypi:~/bro-test $ ls -la
total 28
drwxr-xr-x 3 pi pi 4096 Feb 28 21:03 .
drwxr-xr-x 12 pi pi 4096 Feb 28 20:55 …
-rw-r–r-- 1 root root 699 Feb 28 21:03 conn.log
-rw-r–r-- 1 root root 253 Feb 28 21:03 packet_filter.log
-rw-r–r-- 1 root root 362 Feb 28 21:03 reporter.log
drwx------ 3 root root 4096 Feb 28 21:03 .state
-rw-r–r-- 1 root root 428 Feb 28 21:03 weird.log

On the contrary if on the same machine bro starts with default libpcap i get full network visibility and real traffic logs:

pi@raspberrypi:/opt/bro/logs/current $ ls
capture_loss.log dce_rpc.log dns.log http.log notice.log stats.log stdout.log weird.log
conn.log dhcp.log files.log kerberos.log ssl.log stderr.log syslog.log x509.log
pi@raspberrypi:/opt/bro/logs/current $

pi@raspberrypi:/opt/bro/logs/current $ tail -f conn.log
1488300721.714628 C0ZuNp3n1AAPFqOAP8 fe80::d436:4663:8865:8d25 546 ff02::1:2 547 udp - 62.994834 784 0 S0 F F 0 D 7 1120 0 0 (empty)
1488300873.355133 C4qmHe463s94Z4dwq1 10.0.0.3 43772 10.0.0.1 53 udp dns 0.000407 30 105 SF T T 0 Dd 1 58 1 133 (empty)
1488300873.353585 CHKByP1XMjBTSqRJ7e 10.0.0.3 55014 10.0.0.1 53 udp dns 0.000434 44 107 SF T T 0 Dd 1 72 1 135 (empty)
1488300882.759725 CnoqlW2FFjY3LX3q2 10.0.0.6 49704 10.0.0.5 445 tcp - 14.935617 3786 1209 RSTO T T 0 ShADdaR 13 4318 9 1581 (empty)
1488300864.039916 CwI2Fng0jsi2ODWl9 10.0.0.31 123 193.93.167.241 123 udp - 0.020522 0 48 SHR T F 0 Cd 0 0 1 76 (empty)
1488300874.039921 CN6xHJ2fqNoOloNr4e 10.0.0.31 123 194.116.168.41 123 udp - 0.029709 0 48 SHR T F 0 Cd 0 0 1 76 (empty)
1488300933.675398 Cd7Bql2wn5TNLWBTMg 10.0.0.3 47206 10.0.0.1 53 udp dns 0.000539 30 105 SF T T 0 Dd 1 58 1 133 (empty)
1488300933.674395 CHSnQs2YEeLjiqxOe3 10.0.0.3 55965 10.0.0.1 53 udp dns 0.000245 44 107 SF T T 0 Dd 1 72 1 135 (empty)
1488300889.039915 CALlew1poCyja9ha2l 10.0.0.31 123 91.217.155.60 123 udp - 0.021564 0 48 SHR T F 0 Cd 0 0 1 76 (empty)

So any ideas whats going on ? I couldn’t find any reference of something similar really and i am searching reading and compiling for 2 weeks :slight_smile:

Bro is a great tool and combined with rpi and pf_ring very flexible and powerful in cluster mode. So any help would be highly appreciated to help me with this project. Thanks in advanced

The reporter.log contains errors.. what does it have in it?

pi@raspberrypi:~/bro-test $ cat reporter.log
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path reporter
#open 2017-02-28-21-09-35
#fields ts level message location
#types time enum string string
1488316175.157715 Reporter::INFO received termination signal (empty)
1488316175.157715 Reporter::INFO 674 packets received on interface eth0, 0 dropped (empty)
#close 2017-02-28-21-09-35

ah, well that's not so bad.

The entries that you pasted from your conn.log before only had "^c" for history, which is

        ## ^ connection direction was flipped by Bro's heuristic
        ## c packet with a bad checksum

have you tried bro using the libpcap that comes with pf_ring?

No i haven’t how do i enable it? Just Compile bro with pcap=/opt/pfring/lib/libpcap.so ? It would be faster than standard libpcap but not as fast as pf_ring?

The bad checksum staff is weird but i also tried with -C option with no difference…anyway the whole problem seems pretty unsual

The problem is tha rpi has 100mpbs network card and i want to use cluster and pf_ring and without knowing much seems the best option for real time monitoring on a production network