I have a basic broctl plugin that I’m hoping will be a first step towards the CBrAN outlined here <http://bro.org/development/projects/cban.html>. Currently, most of the functionality is stubbed in as screen output.
My initial thought was for the Bro project to maintain and distribute a SQLite database of rebros (git repos of Bro scripts). As Python2.5+ has native support for SQLite this would not require users to install additional Python modules. A community developer would be able to register his/her rebro for inclusion in the SQLite file distributed with broctl. The SQLite file would represent the universe referred to on the project page.
Before getting too far into the weeds, I’m hoping to get some feedback around design decisions (what issues distributing/maintaining a SQLite file introduces) and to see if anyone is interested in collaborating.
My initial thought was for the Bro project to maintain and distribute a SQLite database of rebros (git repos of Bro scripts).
A terminology nitpick: a "rebro" (according to the proposal doc) isn't a git repo, but a git repo may contain rebros (directories containing a __load__.bro script).
I'm only pointing it out to suggest the first step is that we come to a consensus on what terms to use to describe things else later discussion might get confusing.
Here's things I think need naming:
1) A git repository that may contain bro scripts.
I actually don't think these need a special name, just call it a "repo" or "repository" as usual since there's not really any unique requirements.
2) A directory containing __load__.bro.
These just let Bro "@load <dir>". "rebro" is confusing for this, since it's not a git repo itself. I've always called this a "package" for lack of anything better.
3) CBrAN
Don't think this was actually decided? Works for me, except the case switch in the middle is a bother.
As Python2.5+ has native support for SQLite this would not require users to install additional Python modules. A community developer would be able to register his/her rebro for inclusion in the SQLite file distributed with broctl. The SQLite file would represent the universe referred to on the project page.
If the registration process were to be something simple like a person doing a pull request on GitHub where they've added their repo to the "universe" database, a flat file rather than SQLite might be better for that database file since it should be easier to change and audit (by just looking at the git commit) ?
Though if BroControl parses that file and generates some internal representation that may use SQLite, that might be one way to do it. Were you mostly thinking of storing metadata/dependency stuff there?
Having a name for a directory resembling "repo" when the directory is
not a repository is confusing, I agree. I'll refer to it as a package
moving forward.
The project page refers to the package manager as CBAN, let's go with that.
Having a flat file representation of the universe would make managing
the universe through GitHub much simpler. It would also save people
from having to write a bunch of SQL code.
My thought was for the universe database to contain the following
pieces of information (taken from the project page): name, URL,
author, tags, package version, Bro version, dependencies, license,
description. If a plain text flat file is used would would it be
useful for the file to be readable by the input framework?
Great to see you're up for tackling this, it's one of the major
missing pieces in the Bro world (universe? :).
I like Jon's terminology, and I agree that a flat file will probably
work better than an SQL lite database for storing that meta
information. I don't think the input framework needs to parse it, I
picture all of this being handled outside of Bro itself (i.e., by
BroControl).
I actually need to go back to the project page to remind myself of the
specifics there; it's been a while since that went up and we might
want to update a few things. I'd be happy to chat sometime to flesh
this out further.
I think that sounds ok, though details of how package versioning will work may need some fleshing out up front. Maybe one question to answer first: what level of stability and trustworthiness is expected from the universe?
I'd say a git commit hash for versioning could give some of both. Stability in that external repos can progress as fast as they want, but the universe metadata should still point to a valid version. Trustworthiness in that it's known a universe maintainer reviewed the code associated with the hash. Problem is that a git commit hash is per-repository, but per-package versioning might be needed (i.e. limit of one package per repo w/ this scheme).
I like the idea of associating a commit hash to a version of a
package. I didn't consider that and it's clever. That would make
development of package much more fluid and assure a 'point in time'
version of any package.
My thoughts were to offer zero guarantee around package stability or
trustworthiness to users (other than the package will install,
uninstall, etc with broctl) unless the package is included in the Bro
project. This would prevent the Bro team from having to manage too
much. Package versioning, stability, and questions/support would be
left up to package authors and package trustworthiness would be up to
public reputation. This is generally how community apps are handled
for Splunk.
No matter how versioning is done, an announcement to all package
authors should go out when new version of Bro are cut so to ensure
interoperability.