# Zeek Myricom port aggregation

**URL:** https://community.zeek.org/t/zeek-myricom-port-aggregation/5699
**Category:** Zeek
**Created:** [May 15, 2019, 6:14pm UTC](https://community.zeek.org/t/zeek-myricom-port-aggregation/5699 "2019-05-15T18:14:27Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![greg](https://avatars.discourse-cdn.com/v4/letter/g/71c47a/32.png) [@greg](https://community.zeek.org/u/greg)
#### Post date: [May 15, 2019, 6:14pm UTC](https://community.zeek.org/t/zeek-myricom-port-aggregation/5699/1 "2019-05-15T18:14:27Z")

</div>

Hello,

Hoping someone has some insight into whatever I am doing wrong as try as  
I might, I can't seem to get the Myricom plugin working if configured to  
aggregate port data. Zeek starts and then crashes in every case,  
regardless of configuration ie

interface=myricom::3  
interface=myricom::\*

and snf\_aggregate = T

Here is related dmesg output logged by kdump

[67471.838822] BUG: unable to handle kernel paging request at 00007f0d8459607f  
[67471.838863] IP: [\<ffffffffc0bed569\>] snf\_eop\_ioctl+0x609/0xc60 [myri\_snf]  
[67471.838897] PGD 8000000a93bb9067 PUD 12d142c067 PMD 12d142d067 PTE 8000001d54829025  
[67471.838927] Oops: 0001 [#1] SMP  
[67471.838942] Modules linked in: binfmt\_misc macsec tcp\_diag udp\_diag inet\_diag unix\_diag af\_packet\_diag netlink\_diag myri\_snf(OE) mpt2sas raid\_class scsi\_transport\_sas mptctl mptbase ip6t\_rpfilter ipt\_REJECT nf\_reject\_ipv4 nf\_log\_ipv4 ip6t\_REJECT nf\_reject\_ipv6 nf\_log\_ipv6 nf\_log\_common xt\_LOG xt\_conntrack ip\_set nfnetlink ebtable\_nat ebtable\_broute bridge stp llc ip6table\_nat nf\_conntrack\_ipv6 nf\_defrag\_ipv6 nf\_nat\_ipv6 ip6table\_mangle ip6table\_security ip6table\_raw iptable\_nat nf\_conntrack\_ipv4 nf\_defrag\_ipv4 nf\_nat\_ipv4 nf\_nat nf\_conntrack iptable\_mangle iptable\_security iptable\_raw ebtable\_filter ebtables ip6table\_filter ip6\_tables iptable\_filter dell\_rbu sunrpc dcdbas iTCO\_wdt iTCO\_vendor\_support sb\_edac intel\_powerclamp coretemp intel\_rapl iosf\_mbi kvm\_intel kvm irqbypass crc32\_pclmul joydev  
[67471.839241] ghash\_clmulni\_intel aesni\_intel lrw gf128mul glue\_helper ablk\_helper cryptd mxm\_wmi ext4 mbcache jbd2 pcspkr ipmi\_ssif mei\_me lpc\_ich mei sg ipmi\_si ipmi\_devintf ipmi\_msghandler wmi acpi\_power\_meter ip\_tables xfs libcrc32c sd\_mod crc\_t10dif crct10dif\_generic mgag200 i2c\_algo\_bit drm\_kms\_helper syscopyarea sysfillrect sysimgblt fb\_sys\_fops ttm drm crct10dif\_pclmul crct10dif\_common crc32c\_intel drm\_panel\_orientation\_quirks ahci libahci dca libata tg3 megaraid\_sas ptp pps\_core dm\_mirror dm\_region\_hash dm\_log dm\_mod [last unloaded: myri10ge]  
[67471.839450] CPU: 24 PID: 92952 Comm: bro Kdump: loaded Tainted: G OE ------------ 3.10.0-957.10.1.el7.x86\_64 #1  
[67471.839483] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.9.1 12/04/2018  
[67471.839508] task: ffff95d0e7c41040 ti: ffff95e3197c0000 task.ti: ffff95e3197c0000  
[67471.839531] RIP: 0010:[\<ffffffffc0bed569\>] [\<ffffffffc0bed569\>] snf\_eop\_ioctl+0x609/0xc60 [myri\_snf]  
[67471.839564] RSP: 0018:ffff95e3197c3d38 EFLAGS: 00010006  
[67471.839583] RAX: 0000000000000286 RBX: 0000000000000001 RCX: 0000000000000000  
[67471.839605] RDX: ffff95d0526253d0 RSI: 00007f0d84596000 RDI: ffffb70f589ba7f8  
[67471.839627] RBP: ffff95e3197c3df8 R08: ffffb70f599bb000 R09: 0000000000000003  
[67471.839648] R10: 0000000000000000 R11: 0000000000000000 R12: ffff95d052625000  
[67471.839670] R13: 00007ffeb542d710 R14: 00007ffeb542d710 R15: 0000000000000000  
[67471.839693] FS: 00007f180d6a7900(0000) GS:ffff95eefe900000(0000) knlGS:0000000000000000  
[67471.839717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033  
[67471.839735] CR2: 00007f0d8459607f CR3: 0000001ff663c000 CR4: 00000000003607e0  
[67471.839757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000  
[67471.839778] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400  
[67471.839800] Call Trace:  
[67471.839818] [\<ffffffff98d67ef2\>] ? down\_read+0x12/0x40  
[67471.839840] [\<ffffffffc0bdfba0\>] mx\_common\_ioctl+0x40/0x90 [myri\_snf]  
[67471.839865] [\<ffffffffc0bd44e2\>] mx\_ioctl+0x72/0x290 [myri\_snf]  
[67471.839888] [\<ffffffff98856880\>] do\_vfs\_ioctl+0x3a0/0x5a0  
[67471.839908] [\<ffffffff98d70608\>] ? \_\_do\_page\_fault+0x228/0x500  
[67471.839928] [\<ffffffff98856b21\>] SyS\_ioctl+0xa1/0xc0  
[67471.839947] [\<ffffffff98d75ddb\>] system\_call\_fastpath+0x22/0x27  
[67471.839966] Code: d3 e6 44 85 ce 74 e1 48 83 bf b8 00 00 00 00 75 d1 4c 8b 87 c0 00 00 00 4c 63 d9 41 8b 70 04 48 c1 e6 09 4b 03 b4 dc c0 06 00 00 \<0f\> b6 76 7f 41 39 30 75 b4 4c 89 a7 b8 00 00 00 49 89 bc 24 60  
[67471.840084] RIP [\<ffffffffc0bed569\>] snf\_eop\_ioctl+0x609/0xc60 [myri\_snf]  
[67471.840112] RSP \<ffff95e3197c3d38\>  
[67471.840125] CR2: 00007f0d8459607f

---

<div class="post-metadata">

### Author: ![JustinAzoff](https://avatars.discourse-cdn.com/v4/letter/j/13edae/32.png) [@JustinAzoff](https://community.zeek.org/u/JustinAzoff)
#### Post date: [May 15, 2019, 9:35pm UTC](https://community.zeek.org/t/zeek-myricom-port-aggregation/5699/2 "2019-05-15T21:35:56Z")

</div>

That looks like a bug in the myricom Driver and not zeek. Can you  
reproduce the same kernel issue using tcpdump? You configure  
aggregation for that using SNF\_FLAGS:

SNF\_FLAGS=0x2 (Port aggregation (or merging))  
Flag 0x2 says that the port number that is passed to an application is actually  
a mask of port, not just one port.  
For example, when using tcpdump:  
export SNF\_FLAGS=0x2  
env SNF\_FLAGS=0x2 /path/to/tcpdump -i snf3

Without SNF\_FLAGS=0x2, you would actually try to open snf port 3 (which  
may not exist if you only have one adapter.)

It's possible that you don't need to use aggregation in the first  
place, That is generally only needed if you are connecting a fiber  
tap directly into a card. If flows are being load balanced across  
multiple ports you can just run two different sets of workers, one for  
each port

---

<div class="post-metadata">

### Author: ![greg](https://avatars.discourse-cdn.com/v4/letter/g/71c47a/32.png) [@greg](https://community.zeek.org/u/greg)
#### Post date: [May 15, 2019, 10:04pm UTC](https://community.zeek.org/t/zeek-myricom-port-aggregation/5699/3 "2019-05-15T22:04:40Z")

</div>

tcpdump works perfectly with aggregation, no issues

---

<div class="post-metadata">

### Author: ![greg](https://avatars.discourse-cdn.com/v4/letter/g/71c47a/32.png) [@greg](https://community.zeek.org/u/greg)
#### Post date: [June 5, 2019, 6:32pm UTC](https://community.zeek.org/t/zeek-myricom-port-aggregation/5699/4 "2019-06-05T18:32:18Z")

</div>

Just an update:

I contacted Myricom support about this issue a while back and haven't  
heard anything in a while from them so I believe they are unable to  
duplicate it perhaps, as they generally fix kernel problems very quickly  
in my experience.

Fortunately I will be swapping drives in an array and will need to take  
Zeek down, so I will experiment for a bit before bringing it back up and  
see if I can figure out what the issue is.

This kind of experimentation is very difficult when you don't have a dev  
system to test on. 😛

Greg

---

<div class="post-metadata">

### Author: ![JustinAzoff](https://avatars.discourse-cdn.com/v4/letter/j/13edae/32.png) [@JustinAzoff](https://community.zeek.org/u/JustinAzoff)
#### Post date: [June 5, 2019, 6:39pm UTC](https://community.zeek.org/t/zeek-myricom-port-aggregation/5699/5 "2019-06-05T18:39:43Z")

</div>

Oh, I forgot to send you the recommended configuration for 2 cards in one box…

Most likely you don’t need to be merging the ports… as long as the arista or something is merging the flows for you each port is already getting a consistent subset of flows. At that point the on card aggregation doesn’t do anything for you.

I would use a configuration like this:

[node-foo-card1]  
interface = p1p1  
lb\_method=myricom  
lb\_procs=9  
#check hwloc for numa/pci info  
pin\_cpus=1,3,5,7,9,…

[node-foo-card2]  
interface = p2p1

lb\_method=myricom  
lb\_procs=9

#check hwloc for numa/pci info

pin\_cpus=2,4,6,8,…

the hardest part is using the right pin\_cpus settings. It’s a little easier if you disable HT and then check to see which card is attached to which cpu using hwloc. sometimes it doesn’t matter much, but on some motherboards you can make sure you match up pci slots to physical cpus to avoid moving data between the numa nodes.

---

<div class="post-metadata">

### Author: ![greg](https://avatars.discourse-cdn.com/v4/letter/g/71c47a/32.png) [@greg](https://community.zeek.org/u/greg)
#### Post date: [June 5, 2019, 10:22pm UTC](https://community.zeek.org/t/zeek-myricom-port-aggregation/5699/6 "2019-06-05T22:22:13Z")

</div>

Thanks for that Justin, however I am merging two ports on a single card  
and the reason for doing so is the Arista has 2 x 10G taps and  
aggregating those links within the switch causes drops on the switch due  
to periodic microbursts that exceed the bandwidth to a single 10G tool  
port. The other option of aggregating the traffic in the switch to the  
40G ports as a tool port isn't a viable solution.

This is really vexing because tcpdump has 0 trouble reading the  
aggregated ports, so while the kernel panic error information points to  
the SNF software as a cause; I wonder...

Greg

---

<div class="post-metadata">

### Author: ![greg](https://avatars.discourse-cdn.com/v4/letter/g/71c47a/32.png) [@greg](https://community.zeek.org/u/greg)
#### Post date: [June 5, 2019, 10:30pm UTC](https://community.zeek.org/t/zeek-myricom-port-aggregation/5699/7 "2019-06-05T22:30:35Z")

</div>

Other problem is I am running timemachine on this box, I think it would  
be a little dicey trying to run two separate instances of Zeek and TM -  
not even sure running two instances of TM would work...

---

<div class="post-metadata">

### Author: ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)
#### Post date: [May 6, 2022, 3:46pm UTC](https://community.zeek.org/t/zeek-myricom-port-aggregation/5699/8 "2022-05-06T15:46:30Z")

</div>


