Noticed this today and I believe it’s related to a recent cluster crash. The virtual memory size of the Bro manager continues to grow slowly (56G right now) while the resident active memory remains around 4G. To my knowledge this suggests a memory leak. The system is FreeBSD. We are using a Kafka o…

Are you loading misc/scan or misc/detect-traceroute ?

Oh! You got multiple loggers working? I thought you had the broctl changes right before, but you were seeing those weird issues.. did you figure out what was causing them?

Not really, I was going to reply to the old thread regarding this. I’m in the process of switching back to a single logger and considering trying Kafka export from each worker directly. Right now the logs are backlogged by about 20 minutes which I suspect is the bottleneck issue. Apparently when us…

Do you have any other custom scripts loaded that are using sumstats? With a dedicated logger process the manager doesn't really do anything other than sumstats. Look in your cluster-layout.bro to see what port your manager process is assigned.. with 4 loggers I'd imagine it is around 47765/tcp Th…

No, no custom scripts. I found the cluster overwhelmed again today with massive virtual memory usage and an hour after restarting it the same condition returns. I’m using a single logger since the last time. It seems when using a Kafka only export a single logger works fine as the event timestamps …

What timeframe was that pcap for? If the pcap was from an hour or so, it's probably nothing.. but if that was from a few seconds you could have a problem there. The 2000+ things are definitely related to SSL, as well as the other strings in there.. if you look at the raw tcpdump output those would…

What timeframe was that pcap for? If the pcap was from an hour or so, it’s probably nothing… but if that was from a few seconds you could have a problem there. The pcap is 9 seconds during the problem timeframe at the peak of system usage. The 2000+ things are definitely related to SSL, as well…

…The saga continues… Disabling protocols/ssl/validate-certs relieved the rapid memory leak but the problem still exists over long periods of time. Implemented the change on Feb 6th and by Feb 18th the VZSIZE / RSS was 32G / 2G. By Feb 23rd it was 337G / 24G. FWIW I’m considering migrating the clu…

Logging and memory leak

Zeek

Topic		Replies	Views
Bro cluster requirements and manager logging backlog bug Zeek	34	215	May 6, 2022
Logger Child Memory Leak (logger crashing often) Zeek	5	170	May 6, 2022
Memory Issue with Bro Zeek	29	239	May 6, 2022
Manager swapping.. Zeek	11	160	May 6, 2022
Bro's limitations with high worker count and memory exhaustion Zeek	18	161	May 6, 2022

Logging and memory leak

Related topics