Dfa_state_cache does not follow its max size limit and it can run over this limit quite easily. I am not sure what kind of data are stored in the cache, so I am hesitant to fix the bug. Could you please take a look at it with relation to file analyzer?
I use bro 2.3-397. In one of the protocol analyzers, I call file_mgr->DataIn() and later file_mgr->EndOfFile(). Valgrind shows that if you run it on traffic with a lot of files to analyze, memory builds (and is not freed) at this path:
We had removed the corresponding functionality with one of the earlier
releases already because it was a separate code path that, due to its
performance implications, had to be activated explicitly with a
configure switch, meaning that probably nobody was using it anyways.
It's not really a leak though. While the data structure can keep
growing, the memory remains accessible and the states may be used with
future traffic. That said, depending on the regexps in use, the data
structure can get pretty big over time, and the memory indeed won't be
reclaimed. Snother recent change in preparation for 2.4 was optimizing
the file detetion regexps to cause less such memory usage.