# Broker data store API

**URL:** <https://community.zeek.org/t/broker-data-store-api/4256>\
**Category:** Development\
**Tags:** development\
**Created:** [July 22, 2016, 7:26pm UTC](https://community.zeek.org/t/broker-data-store-api/4256 "2016-07-22T19:26:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Matthias\_Vallentin1](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/matthias_vallentin1/32/596_2.png) [@Matthias\_Vallentin1](https://community.zeek.org/u/Matthias_Vallentin1)\
**Post date:** [July 22, 2016, 7:26pm UTC](https://community.zeek.org/t/broker-data-store-api/4256/1 "2016-07-22T19:26:10Z")

</div>

TL;DR:

&nbsp;&nbsp;- Does anyone use Broker's RocksDB backend?  
&nbsp;&nbsp;- Brief overview of the revamped data store frontend API

I've been working on the Broker data store API a bit, trying to come  
with the smallest denominator possible for an initial release. So far I  
have ported the in-memory SQLite backend over. This made me wonder: did  
anyone ever use (or wanted to use) the RocksDB in production? I wonder  
if we can keep it out for Bro 2.5.

Regarding the API, here's a snippet that illustrates the user-facing  
parts:

&nbsp;&nbsp;// Setup an endpoint.  
&nbsp;&nbsp;context ctx;  
&nbsp;&nbsp;auto ep = ctx.spawn\<blocking\>();

&nbsp;&nbsp;// Attach a master datastore with backend. The semantics of  
&nbsp;&nbsp;// "attaching" are open-or-create: if a master exists under the  
&nbsp;&nbsp;// given name, use it, otherwise create it.  
&nbsp;&nbsp;backend\_options opts;  
&nbsp;&nbsp;opts["path"] = "/tmp/test.db";  
&nbsp;&nbsp;auto ds = ep.attach\<master, sqlite\>("foo", std::move(opts));  
&nbsp;&nbsp;if (!ds)  
&nbsp;&nbsp;&nbsp;&nbsp;std::terminate();

&nbsp;&nbsp;// Perform some asynchronous operations.  
&nbsp;&nbsp;ds-\>put("foo", 4.2);  
&nbsp;&nbsp;ds-\>put(42, set{"x", "y", "z"});  
&nbsp;&nbsp;ds-\>remove(42, "z"); // data at key 42 is now {"x", "y"}  
&nbsp;&nbsp;ds-\>increment("foo", 1.7); // data at key "foo" is now 5.7

&nbsp;&nbsp;// Add a value that expires after 10 seconds.  
&nbsp;&nbsp;ds-\>put("bar", 4.2, time::now() + std::chrono::seconds(10));

&nbsp;&nbsp;// Get data in a blocking fashion.  
&nbsp;&nbsp;auto x = ds-\>get\<blocking\>("foo"); // Equivalent to: get("foo"), the  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// blocking API is the default.

&nbsp;&nbsp;// Get data in a non-blocking fashion. The function then() returns  
&nbsp;&nbsp;// immediately and one MUST NOT capture any variables on the stack by  
&nbsp;&nbsp;// reference in the callback. The runtime invokes the callback as soon  
&nbsp;&nbsp;// as the result has arrived.  
&nbsp;&nbsp;ds-\>get\<nonblocking\>("foo").then(  
&nbsp;&nbsp;&nbsp;&nbsp;[=](const data& d) {  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cout \<\< "data at key 'foo': " \<\< d \<\< endl;  
&nbsp;&nbsp;&nbsp;&nbsp;},  
&nbsp;&nbsp;&nbsp;&nbsp;[=](const error& e) {  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (e == ec::no\_such\_key)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cout \<\< "no such key: foo" \<\< endl;  
&nbsp;&nbsp;&nbsp;&nbsp;}  
&nbsp;&nbsp;});

Here's another setup with two peering endpoints, one having a master and  
one a clone (directly taken from the unit tests). This illustrates how  
data stores and peering go hand in hand.

&nbsp;&nbsp;context ctx;  
&nbsp;&nbsp;auto ep0 = ctx.spawn\<blocking\>();  
&nbsp;&nbsp;auto ep1 = ctx.spawn\<blocking\>();  
&nbsp;&nbsp;ep0.peer(ep1);  
&nbsp;&nbsp;auto m = ep0.attach\<master, memory\>("flaka");  
&nbsp;&nbsp;auto c = ep1.attach\<clone\>("flaka");  
&nbsp;&nbsp;REQUIRE(m);  
&nbsp;&nbsp;REQUIRE(c);  
&nbsp;&nbsp;c-\>put("foo", 4.2);  
&nbsp;&nbsp;std::this\_thread::sleep\_for(propagation\_delay); // master -\> clone  
&nbsp;&nbsp;auto v = c-\>get("foo");  
&nbsp;&nbsp;REQUIRE(v);  
&nbsp;&nbsp;CHECK\_EQUAL(v, data{4.2});  
&nbsp;&nbsp;c-\>decrement("foo", 0.2);  
&nbsp;&nbsp;std::this\_thread::sleep\_for(propagation\_delay); // master -\> clone  
&nbsp;&nbsp;v = c-\>get("foo");  
&nbsp;&nbsp;REQUIRE(v);  
&nbsp;&nbsp;CHECK\_EQUAL(v, data{4.0});

I think this API covers the most common use cases. It's always easy to  
add functionality later, so my goal is to find the smallest common  
denominator.

&nbsp;&nbsp;&nbsp;&nbsp;Matthias

---

<div class="post-metadata">

**Author:** ![Siwek\_Jon](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)\
**Post date:** [July 23, 2016, 5:54pm UTC](https://community.zeek.org/t/broker-data-store-api/4256/2 "2016-07-23T17:54:19Z")

</div>

My recollection is that it was just nice-to-have an optional backend that users could choose, perhaps if they need better performance relative to SQLite. But I probably took the time to try and get that working/ready just as reassurance that the datastore API would be able to implement a variety of backends. Not sure about the choice of RocksDB in particular — could have just been that it happened to pop up on people’s radar at the right time.

Given those historical reasons for it existing, would make sense to me if it were temporarily ignored or removed completely (unless there’s people already invested in using it). Hope that helps.

- Jon

---

<div class="post-metadata">

**Author:** ![Matthias\_Vallentin1](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/matthias_vallentin1/32/596_2.png) [@Matthias\_Vallentin1](https://community.zeek.org/u/Matthias_Vallentin1)\
**Post date:** [July 24, 2016, 6:43pm UTC](https://community.zeek.org/t/broker-data-store-api/4256/3 "2016-07-24T18:43:05Z")

</div>

> Not sure about the choice of RocksDB in particular — could have just  
> been that it happened to pop up on people’s radar at the right time.

It's certainly an industrial-strength key-value, so I think it's solid  
choice for those with better performance when needing persistence.

> Given those historical reasons for it existing, would make sense to me  
> if it were temporarily ignored or removed completely (unless there’s  
> people already invested in using it).

My plan was to put on hold for now, just to have less moving parts. It's  
great that you've already invested the time to understand the API and  
come up with an implementation. Same for SQLite. It took me only a day  
to convert your backend code and read up on SQLite here and there. I  
would imagine it will be the same for RocksDB.

That said, adding backends is fortunately a quite mechanical task. It's  
easy to ship as an incremental release. I'm curious to find out what  
types of backends they would like to see and use once they build  
broker-enabled applications.

&nbsp;&nbsp;&nbsp;&nbsp;Matthias

---

<div class="post-metadata">

**Author:** ![Hosom\_Stephen\_M](https://avatars.discourse-cdn.com/v4/letter/h/d26b3c/32.png) [@Hosom\_Stephen\_M](https://community.zeek.org/u/Hosom_Stephen_M)\
**Post date:** [July 25, 2016, 11:46am UTC](https://community.zeek.org/t/broker-data-store-api/4256/4 "2016-07-25T11:46:20Z")

</div>

I can't speak to whether or not it is 'needed', but I have had desire to use it in the past... The only thing preventing me from doing it was the fact that Broker is currently a fast moving target.

Generally speaking, I was wanting to do it so that I could save state between cluster restarts (specifically for authentication data).

---

<div class="post-metadata">

**Author:** ![Matthias\_Vallentin1](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/matthias_vallentin1/32/596_2.png) [@Matthias\_Vallentin1](https://community.zeek.org/u/Matthias_Vallentin1)\
**Post date:** [July 25, 2016, 3:25pm UTC](https://community.zeek.org/t/broker-data-store-api/4256/5 "2016-07-25T15:25:07Z")

</div>

> I can't speak to whether or not it is 'needed', but I have had desire  
> to use it in the past... The only thing preventing me from doing it  
> was the fact that Broker is currently a fast moving target.

Good to know. Scott Campbell also uses the current version of Broker in  
his projects and mentioned the need for a scalable and performing  
storage backend.

> Generally speaking, I was wanting to do it so that I could save state  
> between cluster restarts (specifically for authentication data).

How many keys to you anticipate in your data store? And what's the rate  
of updates? Any ballpark estimate would be useful here.

Given the interest in a scalable backend, I will bring back support for  
a RocksDB backend.

&nbsp;&nbsp;&nbsp;&nbsp;Matthias

---

<div class="post-metadata">

**Author:** ![Hosom\_Stephen\_M](https://avatars.discourse-cdn.com/v4/letter/h/d26b3c/32.png) [@Hosom\_Stephen\_M](https://community.zeek.org/u/Hosom_Stephen_M)\
**Post date:** [July 25, 2016, 5:41pm UTC](https://community.zeek.org/t/broker-data-store-api/4256/6 "2016-07-25T17:41:05Z")

</div>

The number of key/values would depend on the scale of the environment in the case of the authentication framework. In my last implementation... it was one record per user/host pair... which could scale into the tens of thousands of key/value pairs pretty quickly. I haven't looked at that stuff in a while since I'm eagerly awaiting your rewrite of the Broker APIs 🙂

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\
**Post date:** [May 6, 2022, 3:43pm UTC](https://community.zeek.org/t/broker-data-store-api/4256/7 "2022-05-06T15:43:52Z")

</div>

Closed on May 6, 2022, 3:43pm UTC