Deprecation of &persistent

Zeek
1

Hello all!

TLDR:
I'd like to ask that there be some thought given to the deprecation and
eventual removal of the &persistent option in favor of Broker data
stores. IMHO, there are uses cases where the &persistent attribute is
much more attractive and lower overhead than the data store approach.

Longer:
As you are likely aware, &persistent is now marked deprecated and we
expect it to disappear in the next version or two. The recommendation
for replacement is the much more robust, SQLite backed, Broker data store.

The data store solution is very elegant, though it does seem to require
more fiddling than it ought to to get a data store set up. In the long
term and when dealing with large amounts of data that must be persistent

and synchronized across nodes, this really is a wonderful solution.

That said, there seem to me to be some use cases where that is a massive

hammer to swing at some very small problems. For example, we have one
analysis script that is tracking successful external DNS resolutions.
Specifically, it is keeping track of all IPv4 and IPv6 addresses
resolved in the last 7 days (&read_expire 7 days) in a set. For all
outbound connection attempts, this script generates a notice when the
connection involves an external host that never appeared in a DNS answer

record. This is quite handy when it comes to locating unauthorized

outbound scanning, some C2 behaviors that do not rely on DNS/fast flux
sorts of things, fragile configurations of enterprise services, etc.
This has been performing quite well for several years now in more than
one relatively decent sized networks (100,000+ hosts).

For this problem (and others that I can imagine that would take a
similar tack - i.e., only storing a set, vector, or other single
primitive, rather than a massive record in a table or a table of
tables), the &persistent is perfectly "sized."

Am I alone in thinking that this feature should be retained *along side
of* Broker data stores and potentially documented as recommended for
simple primitive data persistence?

Thanks!

2

I second David’s opinion that some form of a quick-path stores or a new implementation of &persistent should be implemented.

The solution we offer right now makes people write tens of lines of cluster code, learning all details of cluster communication and dealing with a product that’s pretty much impossible to debug (broker). Rinse, repeat, guess why it does not work.

I communicated internally we are not upgrading to 2.6 for now. I scoped the upgrade project to take me half a year (or we would have to take a detection impact).

While it is understood the old &synchronized attribute was an enormous hack, broker gives us the ability to do it right.
An easy to use, transparent attribute or some kind of wrapper is something we should consider to offer.

3

I’m not a core developer so I could be totally incorrect here, but I’ve always been under the impression that &persistent was nearly as broken as &sychronized was, at least in a cluster environment. If I’m correct in my assessment, it’s not really about retaining an existing feature as much as correctly implementing a new feature which I imagine is quite complicated.

I agree the learning curve with Broker and persistent stores is a little steep, but there’s actually a fair bit of good methods for debugging to make sure it’s doing what you think it should be. You also have full control over when it updates the store so your persistence doesn’t rely on a clean shutdown of the nodes.

I imagine you could write some boilerplate code that just fires off a scheduled task via bro_init to update the store, all you’d need to do is paste it in and update the variable name for each policy.

-Dop

4

Some sort of macro system for zeek policies could be useful, to ease use of generic code as suggested by Dop’s comment. Anyone else ever had that thought?