The BPAN project

We've been talking a bit about our Bro community script archive BPAN [1]
quite a bit lately. Below, I summarize how I envision this project to
manifest. Feedback would be much appreciated.

    Matthias

Objectives

    - Facilitate sharing Bro scripts in the community

    - Facilitate the integration of user-provided scripts

    - Specify a set of repositories (or paths inside) to load Bro scripts

Requirements:

    - Sharing scripts should be simple: one should be able to point
      someone to a remote repository and the integration happens
      automatically.

    - Organization by topic tags: similar to package managers, we need a
      way to categorize scripts and bundle them. For example

    - Separation between officially supported and user/experimental
      scripts. For the official ones, we offer support and make sure
      they'll always work with the latest/current version of Bro. We do
      not make such a guarantee for external scripts.

    - Integration with the Input/Output framework. That is, users should
      be able to provide custom data sources and sinks.

Implementation details:

    - github's popular fork-pull-merge model is an excellent fit for
      sharing community repositories. We may want to use
      https://github.com/broids as base for our community scripts. Users
      that want to extend the base simply perform the changes, open a
      pull request, and we can review it. The nice thing is that people
      can comment on it and follow-up changes are nicely reflected.
      Moreover, github automatically lists contributors to a certain
      file, so this encourages coders to share, participate, and get
      "reputation"---akin to stackoverflow.

    - Unlike homebrew, I believe it does not make sense for Bro to have
      single huge repository that contains all scripts and that users
      continuously branch from. Bro users probably want to organize
      their scripts into different repositories and combine multiple
      repositories ad libitum, because some repositories may contain
      sensitive elements and should not be public. Yet, private
      repositories can still be easily integrated with this new
      framework.

      Having separate repositories also keeps the repository size under
      control. Although the input framework will be the main mechanism
      to source external data, I could imagine some folks would like to
      stuff some file-based intelligence in their repositories.

      However, separate repositories require integration efforts, but
      this is manageable in my eyes and outlined below.

    - Directory structure: the directory structure provides implicit
      naming and some meta data. If the directly structure is violated,
      Bro will not be able to automatically load scripts. These are the
      rules:

        * Sub-directories in the repository represent "addressable
          units," e.g., names that users can refer to. I call these from
          now on a "rebro". Each rebro must have at least one file
          called '__load__.bro'. This is the entry point for Bro.

          Example:
            - repo := https://www.github.com/joe-user/bromising/
            - repo/
              repo/a/
              repo/a/__load__.bro
              repo/a/aa.bro
              repo/a/ab.bro
              repo/b/__load__.bro/
              repo/b/x/__load__.bro
              repo/b/x/xx1.bro
              repo/b/x/xx2.bro
              repo/b/y/yy1.bro

          In this example, we have the following rebros:

              repo/a
              repo/b
              repo/b/x

          By placing __load__.bro files judiciously, users can
          explicitly decide what parts of the repo may be interpreted as
          a rebro. We may also allow putting a __load__.bro file in the
          top-level directory.

        * Each __load__.bro file contains additional meta data. Here is
          an example

            ## @version: 0.42
            ## @author: Joe Bro <bro@bro-ids.org>

Requirements:

   - Sharing scripts should be simple: one should be able to point
     someone to a remote repository and the integration happens
     automatically.

Dependencies will likely arise. Particularly problematic are extension scripts which require changes in base scripts that we ship. I don't think we can allow new scripts that require tweaks to base scripts.

   - Organization by topic tags: similar to package managers, we need a
     way to categorize scripts and bundle them. For example

   - Separation between officially supported and user/experimental
     scripts. For the official ones, we offer support and make sure
     they'll always work with the latest/current version of Bro. We do
     not make such a guarantee for external scripts.

I think the clear line between supported/not supported is what we ship with our tarballs.

[1] Here are some other naming suggestions instead of BPAN:

   - Bropane (Bro Policy script Archive NEtwork; inflammable)

+1

       * Each __load__.bro file contains additional meta data. Here is
         an example

           ## @version: 0.42
           ## @author: Joe Bro <bro@bro-ids.org>
           ##
           ## @license: BSD-style (should be the default unless specified)
           ##
           ## @bro: 2.1
           ## @dependencies: b/x, https://my-repo.git/foo
           ##
           ## @tags: tls, x509, browser

Maybe also a short paragraph description of the repo's purpose/goal/function could be included here.

And assuming that, for the officially supported repo, we'll also want auto-generated documentation, note that there's currently some scripting that occurs to take reST markup from README files in the same directory as __load__.bro files and display that in the "index of all packages" along with the link that lets you drill down into individual scripts within. We didn't actually get around to using that anywhere, so it's probably not a big deal to change to using __load__.bro instead, but my main point is that the metadata should get specified in a way that can be rendered in to the docs.

         - The @-prefix inside comments may be used as a specific
           marker for parsing. But any other character would do as
           well.

Maybe we could make this reST-compatible markup to start with for reasons above?

+Jon

Maybe we could make this reST-compatible markup to start with for reasons above?

Makes perfectly sense.

I updated the corresponding project page with your and Adam's feedback:

http://www.bro-ids.org/development/projects/cban.html

    Matthias