We are having some issues with the BRO cluster here @SLAC. I am kind of a noob with respect to BRO and the setup we have @SLAC. Please excuse me and my ignorance.
We have Cisco 3k switch running on tap aggregation mode and it also load-balances traffic to the BRO cluster. We tried to upgrade the switch to the newer NX-OS version but we had some problems and we had to revert to the original version with the exact same configuration.
However, there are no notice.log being generated since the upgrade/downgrade incident. On splunk, the BRO traffic event counts have decreased 1/7th after the incident. I am sure there are things that I am missing after the upgrade/downgrade and I am unable to figure out.
One of the colleague suggested, it might be related to asymmetric flow of forward and reverse packets to the worker nodes which is why BRO is failing to analyze the traffic. So, on the switch, I checked if there is load-balance symmetry command; which is on the switch and I performed tcpdump on bro-worker node and the traffic is communication with bro-manager node.
Planning to involve cisco support tomorrow and to capture traffic from the switchport to Bro worker node and see if I can figure out what’s going on.