# Packet Brick question(s) **URL:** \ **Category:** Development\ **Tags:** development\ **Created:** [November 28, 2016, 4:11pm UTC](https://community.zeek.org/t/packet-brick-question-s/4513 "2016-11-28T16:11:08Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Scott\_Campbell](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@Scott\_Campbell](https://community.zeek.org/u/Scott_Campbell)\ **Post date:** [November 28, 2016, 4:11pm UTC](https://community.zeek.org/t/packet-brick-question-s/4513/1 "2016-11-28T16:11:08Z")
I have been investigating the use of Packet Bricks/netmap as a replacement for pf\_ring on linux, but have a few questions. (1) Is there any documentation except for the git page and the scripts themselves? The script comments are nice and useful, but at times the syntax is rather opaque. (2) Following directions and mailing list recommendations, I have a working version which reads from a heavily loaded 10G ixgbe interface and splits the traffic into 4 netmap interfaces. The script looks like: utilObj:enable\_nmpipes() pe = PktEngine.new("e0", 1024, 8) lb = Brick.new("LoadBalancer", 2) lb:connect\_input("eth6") lb:connect\_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") pe:link(lb) pe:start() where eth6 is the data source interface. Script output looks like: > [root@xdev-w4 PB\_INSTALL]# sbin/bricks -f etc/bricks-scripts/startup-one-thread.lua > [pmain(): line 466] Executing etc/bricks-scripts/startup-one-thread.lua > [print\_version(): line 348] BRICKS Version 0.5-beta > \> utilObj:enable\_nmpipes() > \> pe = PktEngine.new("e0", 1024, 8) > \> lb = Brick.new("LoadBalancer", 2) > \> lb:connect\_input("eth6") > \> lb:connect\_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") > \> pe:link(lb) > [lb\_init(): line 66] Adding brick eth6{0 to the engine > [promisc(): line 96] Interface eth6 is already set to promiscuous mode > 970.328612 nm\_open [444] overriding ARG3 0 > 970.328631 nm\_open [457] overriding ifname eth6 ringid 0x0 flags 0x1 > [netmap\_link\_iface(): line 183] Wait for 2 secs for phy reset > [brick\_link(): line 113] Linking e0 with link eth6 with batch size: 512 and qid: -1 > [netmap\_create\_channel(): line 746] brick: 0xfac090, local\_desc: 0xfac780 > 972.343050 nm\_open [444] overriding ARG3 0 > [netmap\_create\_channel(): line 781] zerocopy for eth6 --\> eth6{0 (index: 0) enabled > [netmap\_create\_channel(): line 786] Created netmap:eth6{0 interface > [netmap\_create\_channel(): line 746] brick: 0xfac090, local\_desc: 0xfac780 > 972.343600 nm\_open [444] overriding ARG3 0 > [netmap\_create\_channel(): line 781] zerocopy for eth6 --\> eth6{1 (index: 1) enabled > [netmap\_create\_channel(): line 786] Created netmap:eth6{1 interface > [netmap\_create\_channel(): line 746] brick: 0xfac090, local\_desc: 0xfac780 > 972.344200 nm\_open [444] overriding ARG3 0 > [netmap\_create\_channel(): line 781] zerocopy for eth6 --\> eth6{2 (index: 2) enabled > [netmap\_create\_channel(): line 786] Created netmap:eth6{2 interface > [netmap\_create\_channel(): line 746] brick: 0xfac090, local\_desc: 0xfac780 > 972.344696 nm\_open [444] overriding ARG3 0 > [netmap\_create\_channel(): line 781] zerocopy for eth6 --\> eth6{3 (index: 3) enabled > [netmap\_create\_channel(): line 786] Created netmap:eth6{3 interface > \> pe:start() and the related dmesg data is: > dmesg: > ixgbe 0000:81:00.0: eth6: detected SFP+: 5 > ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX > 494.566450 [131] ixgbe\_netmap\_configure\_srrctl bufsz: 2048 srrctl: 2 > ixgbe 0000:81:00.0: eth6: detected SFP+: 5 > ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX > 496.743920 [320] netmap\_pipe\_krings\_create ffff880876731a00: case 1, create both ends > 496.744464 [320] netmap\_pipe\_krings\_create ffff880876731000: case 1, create both ends > 496.745026 [320] netmap\_pipe\_krings\_create ffff880878fcb600: case 1, create both ends > 496.745520 [320] netmap\_pipe\_krings\_create ffff880875e06c00: case 1, create both ends > Loading kernel module for a network device with CAP\_SYS\_MODULE (deprecated). Use CAP\_NET\_ADMIN and alias netdev-netmap instead > Loading kernel module for a network device with CAP\_SYS\_MODULE (deprecated). Use CAP\_NET\_ADMIN and alias netdev-netmap instead When I run the "./pkt-gen -i netmap:eth6}0 -f rx" command, I am seeing only a tiny fraction of the expected traffic: > [root@xdev-w4 bin]# ./pkt-gen -i netmap:eth6}0 -f rx > 007.093750 main [2552] interface is netmap:eth6}0 > 007.093855 main [2675] running on 1 cpus (have 32) > 007.094406 extract\_ip\_range [465] range is 10.0.0.1:1234 to 10.0.0.1:1234 > 007.094418 extract\_ip\_range [465] range is 10.1.0.1:1234 to 10.1.0.1:1234 > 007.094481 main [2770] mapped 334980KB at 0x7f325200d000 > Receiving from netmap:eth6}0: 1 queues, 1 threads and 1 cpus. > 007.094525 start\_threads [2235] Wait 2 secs for phy reset > 009.094811 start\_threads [2237] Ready... > 009.094927 receiver\_body [1638] reading from netmap:eth6}0 fd 3 main\_fd 3 > 010.095975 main\_thread [2325] 3.573 Kpps (3.577 Kpkts 2.151 Mbps in 1001085 usec) 511.00 avg\_batch 0 min\_space > 011.097211 main\_thread [2325] 2.552 Kpps (2.555 Kpkts 1.643 Mbps in 1001237 usec) 511.00 avg\_batch 1 min\_space > 012.098314 main\_thread [2325] 3.063 Kpps (3.066 Kpkts 1.981 Mbps in 1001103 usec) 511.00 avg\_batch 1 min\_space ... > ^C021.032505 sigint\_h [512] received control-C on thread 0x7f326672f700 > 021.032531 main\_thread [2325] 2.762 Kpps (2.555 Kpkts 1.306 Mbps in 925126 usec) 511.00 avg\_batch 1 min\_space > 022.033620 main\_thread [2325] 510.000 pps (511.000 pkts 306.248 Kbps in 1001087 usec) 511.00 avg\_batch 1 min\_space > Received 33726 packets 2876632 bytes 66 events 85 bytes each in 12.02 seconds. > Speed: 2.806 Kpps Bandwidth: 1.915 Mbps (raw 2.453 Mbps). Average batch: 511.00 pkts Running all 4 netmap interfaces provides about the same volume of data (which is in total ~ 1% of what I would expect). The cpu usage of the bricks command is ~ 15-20% regardless of running pkt-gen. What can I do differently to get better performance? (3) Accessing the interfaces - as far as actually using the interfaces with bro or tcpdump I am somewhat at a loss. I installed netmap-libpcap version 1.6.0-PRE-GIT\_2016\_11\_26 and compiled tcpdump: [root@xdev-w4 tcpdump-4.6.2]# ./tcpdump --version tcpdump version 4.6.2 libpcap version 1.6.0-PRE-GIT\_2016\_11\_26 OpenSSL 1.0.1e-fips 11 Feb 2013 which is the recommended version per the information on the packet bricks git README. I am unable to get either the native or the netmap-libpcap version of tcpdump to recognize the interfaces: [root@xdev-w4 tcpdump-4.6.2]# ./tcpdump -n -i netmap:eth6{0 tcpdump: netmap:eth6{0: No such device exists (SIOCGIFHWADDR: No such device) The ifconfig info for the interface looks like: > eth6 Link encap:Ethernet HWaddr 00:1B:21:9D:95:EA >           UP BROADCAST RUNNING PROMISC MULTICAST MTU:9000 Metric:1 >           RX packets:39463364964 errors:0 dropped:72437 overruns:0 frame:0 >           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >           collisions:0 txqueuelen:1000 >           RX bytes:56963040769452 (51.8 TiB) TX bytes:0 (0.0 b) At this point I am wondering where to go next. It would be great to use PB instead of pf\_ring, but I will need some help to get there. many thanks! scott ---
**Author:** ![Aashish\_Sharma1](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Aashish\_Sharma1](https://community.zeek.org/u/Aashish_Sharma1)\ **Post date:** [November 28, 2016, 4:37pm UTC](https://community.zeek.org/t/packet-brick-question-s/4513/2 "2016-11-28T16:37:46Z")
Scott, I was using the following script when I was playing with the packet-bricks Dec last year: utilObj = dofile("scripts/utils.lua") utilObj:enable\_nmpipes() pe = PktEngine.new("e0") lb = Brick.new("LoadBalancer", 2) lb:connect\_input("ix0") lb:connect\_output("ix0{1", "ix0{2", "ix0{3", "ix0{4", "ix0{5", "ix0{6", "ix0{7", "ix0{8", "ix0{9", "ix0{10", "ix0{11", "ix0{12", "ix0{13", "ix0{14", "ix0{15", "ix0{16", "ix0{17", "ix0{18", "ix0{19", "ix0{20" ) pe:link(lb) pe:start() pe:show\_stats() This was on FreeBSD. > When I run the "./pkt-gen -i netmap:eth6}0 -f rx" command, I am seeing > only a tiny fraction of the expected traffic: Uncharted terretory for me. I was actually comparing bro logs generated from packet-bricks with the ones on other clusters and bricks vs others seemed reasonably comparable when I last left playing with this. > What can I do differently to get better performance? Others can elaborate more since I haven't tested this at all but I hear netmap support in linux is quite stable now - that might be something to try. Aashish ---
**Author:** ![Asim\_Jamshed](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@Asim\_Jamshed](https://community.zeek.org/u/Asim_Jamshed)\ **Post date:** [November 29, 2016, 6:58am UTC](https://community.zeek.org/t/packet-brick-question-s/4513/3 "2016-11-29T06:58:47Z")
Hi Scott, Please see my answers below: > I have been investigating the use of Packet Bricks/netmap as a > replacement for pf\_ring on linux, but have a few questions. > > (1) Is there any documentation except for the git page and the scripts > themselves? The script comments are nice and useful, but at times the > syntax is rather opaque. At the moment, there is no other documentation. I will try to add more description to the script to explain each line in further detail. > (2) Following directions and mailing list recommendations, I have a > working version which reads from a heavily loaded 10G ixgbe interface > and splits the traffic into 4 netmap interfaces. The script looks like: > > utilObj:enable\_nmpipes() > pe = PktEngine.new("e0", 1024, 8) > lb = Brick.new("LoadBalancer", 2) > lb:connect\_input("eth6") > lb:connect\_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") > pe:link(lb) > pe:start() > > where eth6 is the data source interface. > > Script output looks like: > > > [root@xdev-w4 PB\_INSTALL]# sbin/bricks -f etc/bricks-scripts/startup-one-thread.lua > > [pmain(): line 466] Executing etc/bricks-scripts/startup-one-thread.lua > > [print\_version(): line 348] BRICKS Version 0.5-beta > > \> utilObj:enable\_nmpipes() > > \> pe = PktEngine.new("e0", 1024, 8) > > \> lb = Brick.new("LoadBalancer", 2) > > \> lb:connect\_input("eth6") > > \> lb:connect\_output("eth6{0", "eth6{1", "eth6{2", "eth6{3") > > \> pe:link(lb) > > [lb\_init(): line 66] Adding brick eth6{0 to the engine > > [promisc(): line 96] Interface eth6 is already set to promiscuous mode > > 970.328612 nm\_open [444] overriding ARG3 0 > > 970.328631 nm\_open [457] overriding ifname eth6 ringid 0x0 flags 0x1 > > [netmap\_link\_iface(): line 183] Wait for 2 secs for phy reset > > [brick\_link(): line 113] Linking e0 with link eth6 with batch size: 512 and qid: -1 > > [netmap\_create\_channel(): line 746] brick: 0xfac090, local\_desc: 0xfac780 > > 972.343050 nm\_open [444] overriding ARG3 0 > > [netmap\_create\_channel(): line 781] zerocopy for eth6 --\> eth6{0 (index: 0) enabled > > [netmap\_create\_channel(): line 786] Created netmap:eth6{0 interface > > [netmap\_create\_channel(): line 746] brick: 0xfac090, local\_desc: 0xfac780 > > 972.343600 nm\_open [444] overriding ARG3 0 > > [netmap\_create\_channel(): line 781] zerocopy for eth6 --\> eth6{1 (index: 1) enabled > > [netmap\_create\_channel(): line 786] Created netmap:eth6{1 interface > > [netmap\_create\_channel(): line 746] brick: 0xfac090, local\_desc: 0xfac780 > > 972.344200 nm\_open [444] overriding ARG3 0 > > [netmap\_create\_channel(): line 781] zerocopy for eth6 --\> eth6{2 (index: 2) enabled > > [netmap\_create\_channel(): line 786] Created netmap:eth6{2 interface > > [netmap\_create\_channel(): line 746] brick: 0xfac090, local\_desc: 0xfac780 > > 972.344696 nm\_open [444] overriding ARG3 0 > > [netmap\_create\_channel(): line 781] zerocopy for eth6 --\> eth6{3 (index: 3) enabled > > [netmap\_create\_channel(): line 786] Created netmap:eth6{3 interface > > \> pe:start() > > and the related dmesg data is: > > > dmesg: > > ixgbe 0000:81:00.0: eth6: detected SFP+: 5 > > ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX > > 494.566450 [131] ixgbe\_netmap\_configure\_srrctl bufsz: 2048 srrctl: 2 > > ixgbe 0000:81:00.0: eth6: detected SFP+: 5 > > ixgbe 0000:81:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX > > 496.743920 [320] netmap\_pipe\_krings\_create ffff880876731a00: case 1, create both ends > > 496.744464 [320] netmap\_pipe\_krings\_create ffff880876731000: case 1, create both ends > > 496.745026 [320] netmap\_pipe\_krings\_create ffff880878fcb600: case 1, create both ends > > 496.745520 [320] netmap\_pipe\_krings\_create ffff880875e06c00: case 1, create both ends > > Loading kernel module for a network device with CAP\_SYS\_MODULE (deprecated). Use CAP\_NET\_ADMIN and alias netdev-netmap instead > > When I run the "./pkt-gen -i netmap:eth6}0 -f rx" command, I am seeing > only a tiny fraction of the expected traffic: > > > [root@xdev-w4 bin]# ./pkt-gen -i netmap:eth6}0 -f rx > > 007.093750 main [2552] interface is netmap:eth6}0 > > 007.093855 main [2675] running on 1 cpus (have 32) > > 007.094406 extract\_ip\_range [465] range is 10.0.0.1:1234 to 10.0.0.1:1234 > > 007.094418 extract\_ip\_range [465] range is 10.1.0.1:1234 to 10.1.0.1:1234 > > 007.094481 main [2770] mapped 334980KB at 0x7f325200d000 > > Receiving from netmap:eth6}0: 1 queues, 1 threads and 1 cpus. > > 007.094525 start\_threads [2235] Wait 2 secs for phy reset > > 009.094811 start\_threads [2237] Ready... > > 009.094927 receiver\_body [1638] reading from netmap:eth6}0 fd 3 main\_fd 3 > > 010.095975 main\_thread [2325] 3.573 Kpps (3.577 Kpkts 2.151 Mbps in 1001085 usec) 511.00 avg\_batch 0 min\_space > > 011.097211 main\_thread [2325] 2.552 Kpps (2.555 Kpkts 1.643 Mbps in 1001237 usec) 511.00 avg\_batch 1 min\_space > > 012.098314 main\_thread [2325] 3.063 Kpps (3.066 Kpkts 1.981 Mbps in 1001103 usec) 511.00 avg\_batch 1 min\_space > > ... > > > ^C021.032505 sigint\_h [512] received control-C on thread 0x7f326672f700 > > 021.032531 main\_thread [2325] 2.762 Kpps (2.555 Kpkts 1.306 Mbps in 925126 usec) 511.00 avg\_batch 1 min\_space > > 022.033620 main\_thread [2325] 510.000 pps (511.000 pkts 306.248 Kbps in 1001087 usec) 511.00 avg\_batch 1 min\_space > > Received 33726 packets 2876632 bytes 66 events 85 bytes each in 12.02 seconds. > > Speed: 2.806 Kpps Bandwidth: 1.915 Mbps (raw 2.453 Mbps). Average batch: 511.00 pkts > > Running all 4 netmap interfaces provides about the same volume of data > (which is in total ~ 1% of what I would expect). The cpu usage of the > bricks command is ~ 15-20% regardless of running pkt-gen. > > What can I do differently to get better performance? I will first like to know a little bit more about your setup. I tested packet-brickson linux-3.13.0 kernel (I know it's old.. sorry). Can you please share with me thekernel version as well as the netmap driver version? I want to recreateyour setup in my testbed and see what the problem is. Also, please tryto see how much traffic (bandwidth/pps) you can get without using netmap pipes, i.e.: ./pkt-gen -i netmap:eth6 -f rx I will like to know whether your netmap driver settings are okay. > (3) Accessing the interfaces - as far as actually using the interfaces > with bro or tcpdump I am somewhat at a loss. I installed netmap-libpcap > version 1.6.0-PRE-GIT\_2016\_11\_26 and compiled tcpdump: > > [root@xdev-w4 tcpdump-4.6.2]# ./tcpdump --version > tcpdump version 4.6.2 > libpcap version 1.6.0-PRE-GIT\_2016\_11\_26 > OpenSSL 1.0.1e-fips 11 Feb 2013 > > which is the recommended version per the information on the packet > bricks git README. > > I am unable to get either the native or the netmap-libpcap version of > tcpdump to recognize the interfaces: > > [root@xdev-w4 tcpdump-4.6.2]# ./tcpdump -n -i netmap:eth6{0 > tcpdump: netmap:eth6{0: No such device exists > (SIOCGIFHWADDR: No such device) Were you running packet-bricks when you started tcpdump? Please also try running the following command and see if tcpdump works: ./tcpdump -n -i netmap:eth6 > The ifconfig info for the interface looks like: > > > eth6 Link encap:Ethernet HWaddr 00:1B:21:9D:95:EA > >            UP BROADCAST RUNNING PROMISC MULTICAST MTU:9000 Metric:1 > >            RX packets:39463364964 errors:0 dropped:72437 overruns:0 frame:0 > >            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >            collisions:0 txqueuelen:1000 > >            RX bytes:56963040769452 (51.8 TiB) TX bytes:0 (0.0 b) > > At this point I am wondering where to go next. It would be great to use > PB instead of pf\_ring, but I will need some help to get there. > > many thanks! > scott Regards, --Asim ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:44pm UTC](https://community.zeek.org/t/packet-brick-question-s/4513/4 "2022-05-06T15:44:20Z")