CBAN naming

[Siwek_Jon]

I really don't like calling these things "plugins", nor calling the
whole thing the "plugin manager". I'm with Jan here: I think that
would be quite misleading in terms of what I believe people associate
with "plugin” normally

My argument is that it is true/factual/objective that scripts may used as a form of plugin.

We don’t have any data on exactly what percent of people don’t understand that. If it is a majority, that doesn’t change the fact that the majority has a belief that is not completely true and so it makes sense to go against them in an educational effort.

And that task isn’t even difficult. It takes a single sentence description: “Bro Plugins can be either compiled code, Bro scripts or a combination of both”.

and also with how we've used the term "plugin"
so far. The primary way we've used "plugin" so far is as a compiled,
binary extension. While indeed the structure also accommodates
script-only plugins, that does not warrant calling a set of scripts a
"plugin" in my view. Indeed I don't think most people even realize
that a plugin can be just scripts.

But they are going to be become aware very soon, so isn’t it more awkward to try and continue obfuscating the truth of things?

So my vote goes to "package", "bro-pkg", and "Bro
Package Manager".

The only way I’d be ok with “package” is if it completely replaces all uses of the term “plugin”. Otherwise, I think any other idea so far is better than packages.

- Jon

[Siwek_Jon]

I see what you’re saying, but it’s not in agreement with how the “manager” client has been designed so far: it will be able to manage arbitrary plugins without requiring them to present metadata or be wrapped in any special container. Granted, the presence of metadata allows better forms of management, but there will still be some basic plugin management going on whether or not it exists.

- Jon

[Siwek_Jon]

No, the plugin manager would be able to show all plugins installed through it and also list ones that haven’t been installed. We’ll have to see what the exact commands come out to be, it may end up being the way you assumed.

- Jon

[robin]

But to me a bundle/package is something different: It's a deployment
unit, acting as some sort of container (usually enriched by metadata).
A plugin/script itself could be used with Bro but wrapping it into the
container allows to manage it (e.g. in terms of version, dependencies,
etc.).

That's the way I see it, too.

I'm coming to realize something: maybe the whole confusion comes from
us reusing the plugins' internal structure: we said to reuse the same
directory layout for CBAN (for lack of of a better name because
conveniently we already have a container format that supports both
binary plugins as well as set of standalone scripts. Unfortunately,
that however now leads to conflating terminology, because suddenly
plugins already *are* that container. And that I argue is wrong,
because people writing scripts aren't writing plugins assuming any
common interpretation of that term.

So what if we stepped back and ignored how we will represent/structure
these things internally and just conceptually adapt the model Jan is
describing: we're creating *new* containers that support both scripts
and plugins, and CBAN manages these containers. And then it becomes
an implementation detail how this will exactly look like. If we still
want to reuse the plugin structure, fine. But it would also be ok to
do something different internally if that helps avoid confusion.
Whatever it is, it would only need to make sure that after "install",
everything is layed out correctly for Bro to load it (which for binary
plugins means following their structure at the install location).

(As a side node, bringing "bro -N" into the picture makes things even
more confusing because that's *really* targeting the binary extension
stuff. For example, "-NN" wouldn't show the script code being added).

To me, the important part of a the definition of a plugin for most
software is that it is an externally contributed, self/contained
add-on which extends functionality.

Agree in principle, but "extending functionality" with a plugin is
different that just writing a new Bro script. A "plugin" typically
plugs into a set of hooks that a software provides to extend things it
doesn't provide out of the box; once loaded, that new functionality
then becomes a core feature just as if it had been built in in the
first place. I don't see, e.g., writing a script-level detector for
new vulnerability XYZ like that. If a wrote new Python module
implementing, say, BASE65 encoding, I wouldn't be adding a "plugin" to
Python either.

My argument is that it is true/factual/objective that scripts may used as a form of plugin.

Yes, but per above that's only because we decided to reuse the
internal structure. To me, that's arguing from an implementation-level
artefact, which isn't good starting point for defining terminology.

And that task isn’t even difficult. It takes a single sentence
description: “Bro Plugins can be either compiled code, Bro scripts or
a combination of both”.

Sure, but it's still confusing to tell people you need to write a
plugin to add your new vulnerability detector; whereas so far they
simply wrote a script. Look at the mailing list for how people have
used the term "plugin" so far: I'm pretty sure it's been only about
binary plugins. Nobody writing just a script has said "I have a
question about my plugin".

Robin

[Slagell_Adam_J]

I think we are working from two different mental models. You are thinking of Bro mostly from the programming language perspective and analogies with Python. I am thinking of it as a tool or piece of software. It’s both, but I am not bothered by incongruencies to Python.

Regardless, it seems that you and Jon have irreconcilable differences that eliminate plugin or package. And as the PI and implementer, I give high weight to both of your opinions. Would either of you object to extensions?

So while I *really* hate to do this, I will throw out bro-bee and Bro Extensions for Everyone.

[Siwek_Jon]

So what if we stepped back

Yes, generally we may need to step back. I think this thread started based on the idea that the names of things are entirely subjective and separate from the technical design. But that’s not true — I need to be able to write unambiguous documentation and the choice of using plugins as the top-level container type means that introducing any other name for that container is creating an ambiguity. So yes, we can revisit the choice to use plugins as the top-level container.

and ignored how we will represent/structure
these things internally and just conceptually adapt the model Jan is
describing: we're creating *new* containers that support both scripts
and plugins, and CBAN manages these containers

Yes, that could be a way to do things if I understand it right as:

- New Container
  - Script Container (e.g. what is currently called “script package”)
  - Compiled Code Container (e.g. what is currently called “plugin”)

Another idea is to not have any new container and allow the Script Container and Compiled Code Container to live at the top of the hierarchy as siblings. I think they could still share a common metadata format. It would also still be possible for a person to promote their Script Container to a Compiled Code Container if they find that they need that level of functionality.

(As a side node, bringing "bro -N" into the picture makes things even
more confusing because that's *really* targeting the binary extension
stuff. For example, "-NN" wouldn't show the script code being added).

Yeah, I see how that’s a bit messy now, but skeptical because it might not be much of a fuss to just change/improve the output of a command line argument.

If a wrote new Python module
implementing, say, BASE65 encoding, I wouldn't be adding a "plugin" to
Python either.

Correct, you didn’t write a plugin to Python, but if you had an application called "Foo" that allows users to specify custom encoding schemes via Python scripts, then you wrote a plugin for “Foo”.

So while a Bro script isn’t a plugin for the Bro language, it is a plugin for the Bro application.

A "plugin" typically
plugs into a set of hooks that a software provides to extend things it
doesn't provide out of the box; once loaded, that new functionality
then becomes a core feature just as if it had been built in in the
first place. I don't see, e.g., writing a script-level detector for
new vulnerability XYZ like that.

But writing such a script does extend Bro w/ new functionality that it doesn’t provide out of the box, so it seems to fit your definition of a plugin as well. The only difference I see is that a script is more limited in how far it can extend functionality, but that’s not a binary property you can use to easily categorize plugin vs. non-plugin.

The way I see it: one set of “plugin hooks" that the Bro application provides is the Bro scripting language itself and all the APIs/BIFs/events that are available within it.

My argument is that it is true/factual/objective that scripts may used as a form of plugin.

Yes, but per above that's only because we decided to reuse the
internal structure. To me, that's arguing from an implementation-level
artefact, which isn't good starting point for defining terminology.

But I’m not just arguing based upon the way Bro’s plugins are implemented, I’m arguing about plugins in general. There exists applications that use scripts as a form of plugin.

So I’m saying it’s actually a Good Thing that Bro plugins can be script-only or additionally use compiled code, even if it was not intentional because it fits an even wider description of the term “plugin”.

But if you still don’t like that plugins can be used for scripts-only, then we should probably revisit the design so that we aren’t using them in that way (e.g. the ideas presented near the top of this email).

And that task isn’t even difficult. It takes a single sentence
description: “Bro Plugins can be either compiled code, Bro scripts or
a combination of both”.

Sure, but it's still confusing to tell people you need to write a
plugin to add your new vulnerability detector;

They only need to write a plugin if they plan to distribute it so that it works w/ the new plugin manager tool. I don’t see how that can lead to confusion, but I do see how maybe that could be more tedious than they would like

We can address this problem by designing the manager to seamlessly deal with both script-only containers as well as compiled-code containers like we were thinking about above.

whereas so far they
simply wrote a script. Look at the mailing list for how people have
used the term "plugin" so far: I'm pretty sure it's been only about
binary plugins.

But that doesn’t negate the truth that plugins currently may be either script-only or binary. And those past discussions don’t become invalidated if we move to using script-only plugins more generally.

Nobody writing just a script has said "I have a
question about my plugin”.

Because at the time they said that, they weren’t authoring a plugin, they were writing a script

People don’t currently have a reason to put their scripts inside a plugin, so that’s why no one uses that wording and goes down that path. The incentive for them to start putting their script within a plugin would be the existence of a tool that manages plugins as a top-level container.

I think it may be worth it to do another design pass to see if can find any problems w/ either the super-container or sibling-container approach as outlined at the top. I do like those ideas and only see improvements and no drawbacks (apart from implementation code complexity, which doesn’t count for much), so I don’t mind wrapping my head around it for a bit longer to see if it’s a better fit. Unless I’ve convinced you that “plugin” as the top-level container is the way to go?

- Jon

[Siwek_Jon]

We haven’t had a lot of time to reconcile, but my stance is that it’s not logical to choose a project name that introduces ambiguity in the naming of the top-level containers it deals with.

Some paths to move forward that I see:

1) revisit the design of what the top-level container is

2) use plugins as top-level container and project name that refers to plugins

3) use plugins as top-level container and project name that is totally unrelated to the container name

4) use plugins as top-level container, a project name that refers to them by a different name, and then suggest how docs should be written or re-organized to avoid ambiguity. I’m only being slightly facetious here, I really would need extra help/effort with guidelines for what term to use in what situations. e.g. is it ok to call it a “plugin” when referring directly to existing plugin docs, but I should use the other term otherwise? Seems better to avoid the extra effort and strain this path puts on maintaining consistent/clear docs.

- Jon

[Vlad_Grigorescu1]

Having reread through the discussion, I want to try to take a step back and review some of it.

I believe there are two goals in play:

1. From a user’s perspective, the principle of least astonishment. Names matter, and choosing something intuitive or familiar means we’re not raising the barrier to entry for a new user.
2. From a developer’s perspective, submitting to “CBAN,” maintaining and updating their code should be as painless as possible. Let’s not forget that at the end of the day, the only thing that matters in making this project successful or not is developer contributions. If people aren’t contributing, none of this has any point.

The first goal is the reason we moved away from bags and carts. However, I think we’re in danger of straying too far away from the second goal. I think having containers which manage both scripts and plugins would be a mistake.

A scenario that I see as being quite likely is that a developer starts such a container as being script-only, but wants to expand it at some future date. Say there’s some new botnet with a domain-generation algorithm that they would like Bro to predict and alert on. The script is functional, it’s being deployed, but it has a noticeable performance impact. After some refactoring, the developer adds a BIF to do some string-formatting heavy lifting in C++ instead of in script-land. At this point, how does the script package get converted to a plugin? Is this a new container? Is it a refactoring of the existing container? Will bro-pkg be smart enough to remove the old scripts from the script package and make sure the scripts from the plugin are being loaded?

I fear that in trying to reduce confusion over nomenclature, we’re complicating the design and introducing much more confusion. I believe there are advantages to having script-only plugins. Easier iterative development is one, but also having a well defined structure with a clear place for btest (in fact, init-plugin even creates a simple btest when creating the plugin).

I believe that any issues with the name of “plugin” versus “package” can and will be quickly cleared up when the project is released. I’ll note that the script plugin component is actually the only one currently documented here: https://www.bro.org/sphinx-git/devel/plugins.html

I also think that since many of our developer audience is on this list, and we’ve been spamming them about this for the past 10 days, our user outreach has already happened.

–Vlad

[robin]

- New Container
  - Script Container (e.g. what is currently called “script package”)
  - Compiled Code Container (e.g. what is currently called “plugin”)

Yep, and maybe that script container could then even replace all or
some of the "scripts/" part of the "compiled code container" as well.

Actually, taking this one step further and also back into a circle ...
If we start from this perspective: a new container format that can
carry both scripts and compiled code. Then the *structure* (but not
the *name*) of what's currently a plugin might end up working
actually. Would it solve our issue to simply rename that format from
"plugin" to "container" (or "package" or "bundle" or whatever)

The source format (i.e., what a user writes) would look like this:

    <base>/scripts # Contains Bro script code.
    <base>/src # Contains source for binary plugin
    <base>/tests # Contains tests going with container content.

When installing the plugin, that turns into:

    <install-base>/__bro_container__ # Marks a container.
    <install-base>/scripts # Contains Bro script code.
    <install-base>/lib/<plugin-name>.<os>-<arch>.so # Contains compiled binary plugin.
    <install-base>/lib/bif/ # Contains auto-generated *.bif.bro going with the binary plugin.

We could clean that up a bit for clarity, like renaming to "src/" to
"plugins/" for example. (I wouldn't object to changing the code reading
in the plugins to expect a different layout if that helped solving all
this; that's not cast in stone.)

Another idea is to not have any new container and allow the Script
Container and Compiled Code Container to live at the top of the
hierarchy as siblings.

That would work as well but I think would make it more difficult to
move between the two worlds. I'd prefer a single entity being managed.

So while a Bro script isn’t a plugin for the Bro language, it is a plugin for the Bro application.

Yeah, but Bro *is* the language as well. Anyways, I don't think we'll
agree on this, but that's ok.

Robin

[Matthias_Vallentin1]

A scenario that I see as being quite likely is that a developer starts such
a container as being script-only, but wants to expand it at some future
date. Say there's some new botnet with a domain-generation algorithm that
they would like Bro to predict and alert on. The script is functional, it's
being deployed, but it has a noticeable performance impact. After some
refactoring, the developer adds a BIF to do some string-formatting heavy
lifting in C++ instead of in script-land. At this point, how does the
script package get converted to a plugin?

This scenario seems to be well supported with the "container" layout
that Robin proposed. Since each package/plugin will remain in its own
prefix, it can easily start with a script and later on morph into
full-blown shared library plus a set of bifs.

If we consider everything a self-sufficient container, then even the Bro
base is just one of them. (Certainly a very special one, since it
provides the most functionality.) That reminds of BROPATH. If we loosen
up its semantics, it would refer to a set of prefixes where to search
for containers. For example, consider the following containers:

    /usr/local/share/bro/base
    /usr/local/share/bro/policy
    /usr/local/share/bro/site
    /opt/bro/github-user-foo/bro-fancy-detector
    /opt/bro/github-user-foo/bro-string-algs
    /opt/bro/bar/bro-deep-faf

This would result in the following set of "search paths:"

    /usr/local/share/bro
    /opt/bro/github-user-foo
    /opt/bro/bar

I fear that in trying to reduce confusion over nomenclature, we're
complicating the design and introducing much more confusion. I believe
there are advantages to having script-only plugins.

Ideally, it doesn't matter from a user perspective whether a plugin is
script-only or ships with shared libraries and bifs.

I believe that any issues with the name of "plugin" versus "package" can
and will be quickly cleared up when the project is released.

Yeah, I think either would work. One more thing: we should avoid the
term "container" because there's an entire community emerging where this
term means something very different. I don't think anyone ever proposed
"container" to end up in the terminology, just making sure we're not
going down that road.

    Matthias

[robin]

So Johanna and I just white-boarded a somewhat more generic approach
that we believe would work well to cover the various use-cases without
hardcoding much in terms of structure at all.

The brief summary is that we would allow people to specify how to set
BROPATH and BRO_PLUGIN_PATH, and how to build their binary stuff; and
otherwise don't impose any furher structure on them, just some
defaults.

Here's the idea in more detail:

- We stay with the model of a "container" that can carry both scripts
  and binary plugins. (We could now maybe go back to "package" again;
  but I'll keep using "container" in the following).

- The user's git repository *is* that container, however we don't
  enforce much of a layout on that at all, just some defaults they can
  change if desired. I'll call the disk location of the local clone
  for a user repository <repo>.

- At install time ("cban install" or whatever) <repo> gets copied into
  a subdirectory <name> at a global location <install-base> ("cp -rp
  <repo> <install-base>/<name>"). Uninstallation means removing that
  installation directory.

- For shipping scripts:

    - We simply add <install-base> to Bro's BROPATH. That means one
      can now put "@load <name>" into local.bro and Bro will look for
      a __load__.bro inside the top-level container directory. That
      file can do whatever it wants for loading further scripts
      located anywhere inside the container. One can also load
      individual top-level scripts that way through "@load
      <name>/foo.bro".

    - If one wants to locate the scripts somewhere else inside the
      container, optional meta information can set a different BROPATH
      relative to the top-level directory. For example, to put them
      into "<repo>/scripts", one would specify as meta information
      "bropath=scripts".

- For shipping binary plugins:

    - Through meta information, we let the author specify a build
      command to build all their binary stuff, such as "./configure &&
      make && make test". The command line client runs that command
      inside <repo>.

    - Per default, we expect that build command to create a directory
      <repo>/build that contains a binary plugin. We add
      <install-base>/<name>/build to BRO_PLUGIN_PATH.

    - If one wants to locate the plugin elsewhere, optional meta
      information can set a different BRO_PLUGIN_PATH. For example,
      to put it into "<repo>/compiled/cool_plugin", one would specify
      as meta information "pluginpath=compiled/cool_plugin".

- In addition, we believe we should go back to require that minimal
  set of meta information that we discussed earlier: a container would
  need at least a name, a contact, a version, and a licence. For the
  optional meta information described above, we preset defaults:

    bropath=.
    pluginpath=./build
    buildcmd= # Empty means: nothing to build.

So, examples:

    1. A container with just a single script pulled in through "@load foo".

       <repo>/META
            name=foo
            version=1.0
            license=bsd
            contact=foo@.bar.com

       <repo>/__load_.bro
            event bro_init() { print "Foo!"; }

    2. A container with multiple scripts all loaded through "@load foo".

       <repo>/META
            name=foo
            version=1.0
            license=bsd
            contact=foo@bar.com

       <repo>/__load_.bro
            @load ./script1
            @load ./script2

       <repo>/script1.bro
            event bro_init() { print "Foo1"; }

       <repo>/script2.bro
            event bro_init() { print "Foo2"; }

    3. Same as (2), but the scripts now in a subdirectory scripts/:

       <repo>/META
            name=foo
            version=1.0
            license=bsd
            contact=foo@bar.com
            bropath=scripts

       <repo>/scripts/__load_.bro
       <repo>/scripts/script1.bro
       <repo>/scripts/script2.bro

    4. A binary plugin:

       <repo>/META
            name=foo
            version=1.0
            license=bsd
            contact=foo@bar.com
            buildcmd=./configure && make && make test

       <repo>/configure
       <repo>/src/*
       <repo>/tests/*

       (The skeleton that init-plugin creates works for this directly
       already, except for the meta information.)

    5. A binary plugin with different build command and location:

       <repo>/META
            name=foo
            version=1.0
            license=bsd
            contact=foo@bar.com
            buildcmd=build-all-my-stuff.sh
            pluginpath=compiled/cool_plugin

       <repo>/build-all-my-stuff.sh
       <repo>/<any-other-stuff-that-the-shell-scripts-needs>

[Siwek_Jon]

2) From a developer's perspective, submitting to "CBAN," maintaining and updating their code should be as painless as possible. Let's not forget that at the end of the day, the only thing that matters in making this project successful or not is developer contributions. If people aren't contributing, none of this has any point.

The first goal is the reason we moved away from bags and carts. However, I think we're in danger of straying too far away from the second goal. I think having containers which manage both scripts and plugins would be a mistake.

Yeah, we do need to keep the goal of making things easier for devs/authors, but I’m wondering if having the top-level containers be plugins doesn’t actually make things a harder for people who were only writing scripts to begin with?

They now have to learn what a plugin is and how to make one instead of just being able to ship their directory full of scripts and have it work. Let me know what you think.

A scenario that I see as being quite likely is that a developer starts such a container as being script-only, but wants to expand it at some future date.

I’m following Matthias on this in that I see such a super-container at the top-level being able to deal w/ changing from script-only to scripts+compiled in a way that’s transparent to users. We may also want a form a pinning that allows a user to say “don’t upgrade this if it changes from script-only to compiled code”.

You had some good implementation detail questions about how to make this work well that I think all have answers, so in interest of keeping the thread shorter I won’t go in to them now. But let me know if you see one problem that will be particularly tricky that we should go into detail about.

I believe that any issues with the name of "plugin" versus "package" can and will be quickly cleared up when the project is released. I'll note that the script plugin component is actually the only one currently documented here: https://www.bro.org/sphinx-git/devel/plugins.html

Yeah, I’ll concede maybe people will get used to it, but I at least need to be working from a design spec that tells me exactly what to do concerning that page. If we start using “package” in one place I will eventually need to start talking about how a “plugin” works and refer to that page, which is currently a user-facing doc. Do canonicalize it to use “package”? Do I leave it alone and just have a non-sequitur transition in talking about “packages” and then suddenly “plugins” ?

If we switch the design to instead user the super-container format, then that’s not an issue for me anymore because the relationship changes from “is a plugin” to “may have a plugin”.

- Jon

[Daniel_Thayer]

Could you clarify what you mean by BRO_PLUGIN_PATH here? Are you saying
that after I do a "cban install cool-plugin", I'd need to manually
set an environment variable in order for Bro to find the new plugin?
Or are we still planning to default to <PREFIX>/lib/bro/plugins/ ?

[robin]

Could you clarify what you mean by BRO_PLUGIN_PATH here?

I mean we magically extend the BRO_PLUGIN_PATH so that Bro will find
it there. No manual configuration, CBAN will take care of pointing Bro
to the right place inside the plugin's installation directory. (And
we'd need to add mechanism to Bro to actually do that.)

Or are we still planning to default to <PREFIX>/lib/bro/plugins/ ?

That default location will stay, we just add the new ones to the
internal search path as well.

Robin

[johanna]

No, you do not, that happens automatically. Users also have the ability to
specify a subdirectory inside of their repository that is then used as the
BRO_PLUGIN_PATH. That would be done by the plugin authors if they want to
change the directory structure. For the installing user, everything
remains automatic.

Johanna

[Siwek_Jon]

- For shipping scripts:

   - We simply add <install-base> to Bro's BROPATH. That means one
     can now put "@load <name>" into local.bro and Bro will look for
     a __load__.bro inside the top-level container directory. That
     file can do whatever it wants for loading further scripts
     located anywhere inside the container. One can also load
     individual top-level scripts that way through "@load
     <name>/foo.bro”.

Similar to Daniel’s question, is there a one time setup the user does or they need to modify local.bro every time they install a new container?

I was thinking there’s a directory full of scripts from containers and Bro automatically loads all of them. If a user did “cban install <name>” then that container’s scripts are all auto-loaded. But if a user did “cban unload <name>” then they are free to still manually load scripts from it, but otherwise it’s not auto-loaded. They can then do “cban load <name>” to switch back to a default load-everything behavior.

- In addition, we believe we should go back to require that minimal
set of meta information that we discussed earlier: a container would
need at least a name, a contact, a version, and a license.

Is that metadata a requirement for submission to the community repo or just for general interoperability with the container manager client?

From user perspective, the later may be fine except potentially in the use-case where a person has no plans to submit their container in the near-term, but want to use the manager client and the container structure maybe for sake of consistency or other reasons?

The other potential problem is if you require metadata and then call that container a “package”, we are back to the issue of me needing to know what to do about the current use of the term “package” which refers to a collection of scripts that require no metadata. A “package” can't require metadata and also not require it at the same time.

If it doesn’t confuse users to rename the current “package,” that seems like an ok path to go forward with. A rename may at least temporarily confuse/bother me because I’ve used the term “package” consistently in the past, but I’m not a user myself and don’t know whether the usage is common among actual users, so I need someone else to decide.

- Jon

[robin]

Similar to Daniel’s question, is there a one time setup the user does
or they need to modify local.bro every time they install a new
container?

In this case I was thinking modify local.bro. That said, I remember
your "load"/"unload" now, that would indeed be an alternative. What
let's me hesitate a bit about that is that it doesn't match how people
interact with scripts currently. Right now they have to @load them
explicitly (unless they are in base/, where however we don't expect
user scripts).

Also, how would this work when using Bro from the command line (in
contrast to BroControl)? Would it still pull in all the modules that
have been loaded through the client? Could that cause confusion
because we wouldn't have a standardized setting anymore? Would there
be a way to prevent it?

Is that metadata a requirement for submission to the community repo or
just for general interoperability with the container manager client?

Mostly for submission, though local installation through the client
would need some as well I think (the name[1] and the paths at least).
But people could always run out out of their local clone directly by
setting BROPATH and BRO_PLUGIN_PATH manually (which is probably a good
way for development anyways).

Johanna suggested that if we do something similar to init-plugin to
create a skeleton container, it would just put a meta file in place
with defaults for the small set of fields we require for submission,
so it wouldn't be much of a hassle either way.

[1] Actually this occurred to me while writing the earlier mail: I
think we need to let people set the name rather than just using the
git repository name (which I at least had in mind so far). They may
want to name their repository differently then what's the container
should be called.

The other potential problem is if you require metadata and then call
that container a “package”,

I'm not attached to using "package". "bundle" works for me as well, as
I think "container" does (though Matthias has a point there about
meanings uses of "container"). I just don't like "plugin".

Robin

[Siwek_Jon]

Similar to Daniel’s question, is there a one time setup the user does
or they need to modify local.bro every time they install a new
container?

In this case I was thinking modify local.bro. That said, I remember
your "load"/"unload" now, that would indeed be an alternative. What
let's me hesitate a bit about that is that it doesn't match how people
interact with scripts currently. Right now they have to @load them
explicitly (unless they are in base/, where however we don't expect
user scripts).

So scripts do not autoload, but plugins do? I’m thinking the procedure a user takes to get things working should be the same regardless of script-only vs. binary plugin.

And if the process isn’t the same in both cases, is that also in conflict w/ the goal of a developer being able to promote a script-only thing into a binary plugin without users noticing?

Also, how would this work when using Bro from the command line (in
contrast to BroControl)? Would it still pull in all the modules that
have been loaded through the client? Could that cause confusion
because we wouldn't have a standardized setting anymore? Would there
be a way to prevent it?

I was thinking that Bro command line and BroControl would have to work similarly. So it may be Bro needs a direct change to how it handles auto-loading everything within a directory, either hardcoded specifically for use with the new containers or else a path specified by a new environment variable?

The other potential problem is if you require metadata and then call
that container a “package”,

I'm not attached to using "package". "bundle" works for me as well, as
I think "container" does (though Matthias has a point there about
meanings uses of "container"). I just don't like "plugin".

I agree that plugin no longer matches this design when referring to top-level container. And package maybe has less severe conflicts than before, but I'd still need some clarification in how to use it within code/docs. If package is still desirable, then maybe try to silently change current usage of “script packages” into just “scripts” without a new container name at all for them. I also suggest “script directory” as a potential name if needed. Then reappropriate “package" for use in this new project.

To mock up example documentation:

"
The Bro Package Manager is a tool that makes it easy for Bro users to install and manage third party scripts and binary plugins. It does this by providing both a communal GitHub repository where developers contribute packages they have created and a command line tool, ‘bro-pkg', for users to fetch packages from that repository. The packages that users submit may contain either scripts or binary plugins for Bro and the command line client automatically handles their installation process and interdependencies with other packages.

To create a package, a developer needs to

1) Create a directory and metadata file with the following required fields... (TODO: details)

2) For packages that just contain Bro scripts, see the following docs about Bro’s script loading process (TODO: there’s maybe not a good doc on the script loading process besides [1] and [2], so maybe need a better one) and then fill out these additional metadata fields... (TODO: details)

3) For packages that contain binary plugins for Bro, see the following docs about plugins: [3]. Then fill out these additional metadata fields... (TODO: details)

4...) (TODO: get into submission process, maybe talk about btest for unit tests and other ways to improve package quality ratings, etc)

“

That all looks consistent except part (2) ends up pointing people toward existing docs/examples that reference “package” but with a different meaning. I'd need a decision to be made about how/whether to change that.

If people are happy w/ the new design proposal and ready to revisit naming, it may be helpful to plug in (pun intended) their ideas for naming the project/client/containers as substitutes for what I used in the above example or make a new, similar one and read it through to see if it’s still coherent.

- Jon

[1] Bro Package Index — Bro 2.6.1 documentation
[2] https://www.bro.org/sphinx/scripting/index.html
[3] https://www.bro.org/sphinx-git/devel/plugins.html

[robin]

So sounds like this proposal is something you can agree with?

So scripts do not autoload, but plugins do?

Thinking more about that I would answer: yes. Going back to the
principle of least surprise, this is how it is now as well: scripts
need to be loaded, plugins load automatically. I suggest we start like
that, we can always add an auto-load mechanism later if it turns out
that would be useful.

And if the process isn’t the same in both cases, is that also in
conflict w/ the goal of a developer being able to promote a
script-only thing into a binary plugin without users noticing?

Isn't that more about moving some functionality, like bifs? I don't
think a plugin would replace the scripts completely.

To mock up example documentation:

I like this, with the one note that most likely they won't need to
fill out any additional meta data fields in (2) and (3) because the
defaults will do.

That all looks consistent except part (2) ends up pointing people
toward existing docs/examples that reference “package” but with a
different meaning. I'd need a decision to be made about how/whether
to change that.

Ok, then let's rename "package" in the current docs. I propose
"module" as the replacement: it's not quite right regarding the
language's module concept but close enough I would say.

naming the project/client/containers

My vote: Bro Package Manager, bro-pkgs, package.

Robin

[Slagell_Adam_J]

That all looks consistent except part (2) ends up pointing people
toward existing docs/examples that reference “package” but with a
different meaning. I'd need a decision to be made about how/whether
to change that.

Ok, then let's rename "package" in the current docs. I propose
"module" as the replacement: it's not quite right regarding the
language's module concept but close enough I would say.

Or “bundle", which has even less baggage, but I think module is fine.

naming the project/client/containers

My vote: Bro Package Manager, bro-pkgs, package.

Only difference is I don’t think the client needs the plural. I would simply go to bro-pkg.