Out of a discussion with Seth and Vlad this morning, I put togehter a
project description for integrating Bro with osquery as a host-based
sensor, using Broker for communication.
https://www.bro.org/development/projects/osquery.html
It's just a first stab, feedback welcome.
Robin
That’s a really nice summary. Thanks!
Also, I spent a bit of time digging through the osquery source yesterday and it looks like it’s possible with the api they expose to submit new queries into osqueryd dynamically so that we could just start up osqueryd and Bro would send over all of the queries that we would like the host to run.
.Seth
+ - On the osquery side, we need to assemble the event for sending
+ to Broker. Generally, the columns returned by the ``SELECT``
+ will turn into the event's arguments. In addition, we add an
+ always-present ``h: Host`` argument. The event arguments' types
+ need to be mapped from what osquery returns to Broker types
+ (which, in turn, correspond to Bro types); see next bullet.
+
+ - It seems there are two possible ways of doing the type conversion:
+
+ 1. Hardcoding: The osqery plugin retrieves the query response,
+ iterates through its columns and builds up a Broker event
+ to then send out.
+
+ .. note::
+
+ I'm not quite sure what interface(s) osquery provides
+ for extracting results. On the web page, I see JSON; not
+ sure if there's something more direct.
+
+ 2. Leveraging JSON: We can also extend Broker with a JSON
+ interface, so that the osqery plugin can forward a JSON
+ response directly. For this, we would:
+
+ - Extend Broker's API with a function that builds an
+ event from JSON; with some predefined mapping of how
+ JSON values turn into Broker values.
+
+ - Then call that function from the osquery plugin.
+
+ Option (2) would actually be a nice interface for Broker to
+ have anyways, as it opens it up to ingesting input from a
+ variety of other JSON sources as well (we could write a an
+ ingestion daemon that opens up a socket to which web
+ applications can post JSON; but that's a different topic :).
I’m not sure what the difference between (1) and (2) is? Either one seems to do a JSON -> Broker-data conversion, the difference is just in whether that conversion code lives in the application that uses Broker or in the Broker library itself. I don’t think Broker itself is in any better position to actually do the conversion. Not opposed to putting such a example/template in Broker, just saying it may not be required to get the job done.
A third idea: it seems like here it would be doing a JSON -> Broker-data -> Bro-value conversion, instead can Broker messages/events just be specified in terms of a JSON string parameter, then leave JSON -> Bro-value conversion up to Bro? Teaching Bro a good way to interface directly w/ JSON might also be beneficial in other areas.
- Jon
I’m not sure what the difference between (1) and (2) is? Either one
seems to do a JSON -> Broker-data conversion, the difference is just
in whether that conversion code lives in the application that uses
Broker or in the Broker library itself.
Correct if JSON is the only way to get data out of osquery. That's a
part I don't know, there might a more direct programmatic interface in
osqyery that doesn't go through JSON.
But let's say we need or want to go through JSON. Then indeed, the
question is where the code lives. Broker isn't in better spot to do
the conversion, but if it were in there, it could be reused by other
data sources than osquery; vs., if it's part of the osquery plugin,
nobody else would benefit from it.
It could also be part of the osquery side initially, and we'd move it
over later if demand turns out to be there.
A third idea: it seems like here it would be doing a JSON ->
Broker-data -> Bro-value conversion, instead can Broker
messages/events just be specified in terms of a JSON string parameter,
then leave JSON -> Bro-value conversion up to Bro?
Yeah, JSON input is on Seth's Bro wishlist. 
But I don't like this
model here because it feels like it's using Broker just a transport
mechanism for raw data. I think the better general approach is to fit
external data into Broker's data model, because then any Broker node
can work with the data, not just those that happen to know how to
interpret the blob coming in.
Robin
It could also be part of the osquery side initially, and we'd move it
over later if demand turns out to be there.
That’s more what I was thinking. Either way doesn’t seem like a huge deal to me: don’t expect the code involved to be that tricky.
A third idea: it seems like here it would be doing a JSON ->
Broker-data -> Bro-value conversion, instead can Broker
messages/events just be specified in terms of a JSON string parameter,
then leave JSON -> Bro-value conversion up to Bro?Yeah, JSON input is on Seth's Bro wishlist.
model here because it feels like it's using Broker just a transport
mechanism for raw data. I think the better general approach is to fit
external data into Broker's data model, because then any Broker node
can work with the data, not just those that happen to know how to
interpret the blob coming in.
Yeah, if there’s many disparate applications acting as nodes here, then may be better to use Broker’s data as common format to ensure everyone has the tools necessary to interpret the messages.
- Jon
deal to me: don’t expect the code involved to be that tricky.
Yep, indeed.
Yeah, if there’s many disparate applications acting as nodes here,
then may be better to use Broker’s data as common format to ensure
everyone has the tools necessary to interpret the messages.
(I won't claim that that's necessarily the case here, but I think it's
good to establish a precedent that this is the right way to do it.)
Robin
Huh, that’s actually a good point. Not quite sure how that would look yet though.
Also, I when I was digging around in osquery, their default view of data internally seems to be in a plist-type format. They have a routine that converts it to json for output. So we certainly aren’t bound to json with this in any way.
.Seth