# Packet Loss with Myricom Bro Cluster **URL:** \ **Category:** Zeek\ **Created:** [April 24, 2017, 10:15pm UTC](https://community.zeek.org/t/packet-loss-with-myricom-bro-cluster/4801 "2017-04-24T22:15:00Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Rogers\_Zach](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Rogers\_Zach](https://community.zeek.org/u/Rogers_Zach)\ **Post date:** [April 24, 2017, 10:15pm UTC](https://community.zeek.org/t/packet-loss-with-myricom-bro-cluster/4801/1 "2017-04-24T22:15:00Z")
Hello, I have a cluster of five machines for our current Bro configuration: One is acting as the Master & Proxy node, and the other four are Worker nodes. The four worker nodes have Myricom 10g NICs and are using the Bro::Myricom plugin. The workers are getting traffic from the Myricom NICs, though there seems to be substantial packet loss. Here is the output of running 'broctl netstats all', as you can see the dropped packet count is quite high:     manager: 1493071687.452948 recvd=5089239 dropped=15719766 link=20825230     proxy-1: 1493071730.166046 recvd=0 dropped=0 link=0 worker-1-1: 1493071533.397463 recvd=16723822 dropped=1827590608 link=1844314430 worker-1-10: 1493071533.598767 recvd=15599277 dropped=1827590608 link=1843189885 worker-1-11: 1493071533.799606 recvd=15290549 dropped=1827590608 link=1842881157 worker-1-12: 1493071534.001465 recvd=17829130 dropped=1827590608 link=1845419738 worker-1-2: 1493071534.201668 recvd=16913145 dropped=1827590608 link=1844503753 worker-1-3: 1493071534.402914 recvd=13317747 dropped=1827590608 link=1840908355 worker-1-4: 1493071534.603851 recvd=16654854 dropped=1827590608 link=1844245462 worker-1-5: 1493071534.791954 recvd=15707229 dropped=1827590608 link=1843297837 worker-1-6: 1493071535.005706 recvd=44924122 dropped=1827590608 link=1872514730 worker-1-7: 1493071535.206737 recvd=15892468 dropped=1827590608 link=1843483076 worker-1-8: 1493071535.407748 recvd=15109664 dropped=1827590608 link=1842700272 worker-1-9: 1493071535.609068 recvd=17725999 dropped=1827590608 link=1845316607 worker-2-1: 1493071452.728863 recvd=12467133 dropped=1654102950 link=1666570083 worker-2-10: 1493071452.930009 recvd=18175352 dropped=1654102950 link=1672278302 worker-2-11: 1493071453.129614 recvd=14050575 dropped=1654102950 link=1668153525 worker-2-12: 1493071453.330973 recvd=13461021 dropped=1654102950 link=1667563971 worker-2-2: 1493071453.532331 recvd=18143473 dropped=1654102950 link=1672246423 worker-2-3: 1493071453.731385 recvd=21263867 dropped=1654102950 link=1675366817 worker-2-4: 1493071453.933757 recvd=21217186 dropped=1654102950 link=1675320136 worker-2-5: 1493071454.135406 recvd=15458489 dropped=1654102950 link=1669561439 worker-2-6: 1493071454.336124 recvd=27904731 dropped=1654102950 link=1682007681 worker-2-7: 1493071454.535881 recvd=15498799 dropped=1654102950 link=1669601749 worker-2-8: 1493071454.738093 recvd=15570231 dropped=1654102950 link=1669673181 worker-2-9: 1493071454.941583 recvd=15546579 dropped=1654102950 link=1669649529 worker-3-1: 1493071595.728039 recvd=16016236 dropped=1393679752 link=1409695988 worker-3-10: 1493071595.928627 recvd=16806524 dropped=1393679752 link=1410486276 worker-3-11: 1493071596.129858 recvd=16960142 dropped=1393679752 link=1410639894 worker-3-12: 1493071596.331357 recvd=19031806 dropped=1393679752 link=1412711558 worker-3-2: 1493071596.531622 recvd=16059903 dropped=1393679752 link=1409739655 worker-3-3: 1493071596.732314 recvd=20528301 dropped=1393679752 link=1414208053 worker-3-4: 1493071596.933870 recvd=17478728 dropped=1393679752 link=1411158480 worker-3-5: 1493071597.134664 recvd=15785290 dropped=1393679752 link=1409465042 worker-3-6: 1493071597.335570 recvd=13051546 dropped=1393679752 link=1406731298 worker-3-7: 1493071597.529927 recvd=17668246 dropped=1393679752 link=1411347998 worker-3-8: 1493071597.736828 recvd=21655950 dropped=1393679752 link=1415335702 worker-3-9: 1493071597.938700 recvd=18237026 dropped=1393679752 link=1411916778 worker-4-1: 1493071949.358339 recvd=19489052 dropped=1504105858 link=1523594910 worker-4-10: 1493071949.559733 recvd=18684571 dropped=1504105858 link=1522790429 worker-4-11: 1493071949.760921 recvd=18578400 dropped=1504105858 link=1522684258 worker-4-12: 1493071949.961847 recvd=15738360 dropped=1504105858 link=1519844218 worker-4-2: 1493071950.160646 recvd=17092444 dropped=1504105858 link=1521198302 worker-4-3: 1493071950.363595 recvd=16890327 dropped=1504105858 link=1520996185 worker-4-4: 1493071950.564875 recvd=14940510 dropped=1504105858 link=1519046368 worker-4-5: 1493071950.765612 recvd=17175752 dropped=1504105858 link=1521281610 worker-4-6: 1493071950.966709 recvd=14854076 dropped=1504105858 link=1518959934 worker-4-7: 1493071951.167789 recvd=15455452 dropped=1504105858 link=1519561310 worker-4-8: 1493071951.368743 recvd=14915366 dropped=1504105858 link=1519021224 worker-4-9: 1493071951.572223 recvd=21105493 dropped=1504105858 link=1525211351 I am looking for guidance on how to figure out the cause of this packet loss, with hopes of fixing the issue. Any ideas? All the best, ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [April 24, 2017, 10:36pm UTC](https://community.zeek.org/t/packet-loss-with-myricom-bro-cluster/4801/2 "2017-04-24T22:36:56Z")
> Hello, > > I have a cluster of five machines for our current Bro configuration: One > is acting as the Master & Proxy node, and the other four are Worker > nodes. The four worker nodes have Myricom 10g NICs and are using the > Bro::Myricom plugin. > > The workers are getting traffic from the Myricom NICs, though there > seems to be substantial packet loss. Here is the output of running > ‘broctl netstats all’, as you can see the dropped packet count is quite > high: > worker-1-12: 1493071534.001465 recvd=17829130 dropped=1827590608 > link=1845419738 … > I am looking for guidance on how to figure out the cause of this packet > loss, with hopes of fixing the issue. > > Any ideas? There’s an issue with the myricom interface where the dropped packet counter is per card and it does not reset to 0 when the application restarts. This causes the dropped packet counter to appear to be (in your case) at least 12x larger than it is… If not more, depending on the uptime of the server. Run this to reset the dropped counter and restart bro: broctl stop broctl exec /opt/snf/bin/myri\_counters -c -p 0 broctl exec /opt/snf/bin/myri\_counters -c -p 1 broctl start That will reset all counters to zero before bro starts and ensure the drop count isn’t crazy high before bro even starts. I wrote the attached script a while ago to compute the proper per card drop percentage. It should still work. Some of these issues can possibly be fixed inside of bro/broctl. I think bro can just grab the drop count at startup and if it’s non-zero use it as an offset for any future values. The drop count being per card is a bit harder. You could just divide the drop count by the number of workers, but that’s really just trading one misleading value for another. [netstats\_sum.py](https://community.zeek.org/uploads/short-url/J8grHtwbb0MXrzdgkuDeZScwhJ.py) (785 Bytes) ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:44pm UTC](https://community.zeek.org/t/packet-loss-with-myricom-bro-cluster/4801/3 "2022-05-06T15:44:52Z")