So this has become an involved thread. Do we need a call, or do you think you can pull out enough to get started Jon?
I also think going with a hierarchical organization like that would help the situation you describe — if an original author of a plugin becomes unresponsive then another person who wants to start maintaining it could “fork” and submit that to cban, Then hopefully when a cban user is searching around it would be apparent that a submission got forked by other users in the community and the answer to why and which version they should use is a few “cban info <repo>” commands away.
- Jon
More generally, there will presumably some functionality to add
"remotes" to one's configuration, allowing plugin writers to point to
experimental code if they wish. Then they can still hack out code and
mix it with existing CBAN plugins, at their own risk.
Yes, part of the plan is to allow one to add arbitrary git repos. i.e. repos that haven’t formally been submitted to CBAN, private ones, etc.
On a cosmetic note, will thing be called CBAN? I find it a very cryptic
name, often confused it with BPAN (even though it doesn't make sense),
and was wondering whether we should converge on some more pronounceable
candidates.
I’m just using CBAN since that seems like the name that stuck around longest without anything better being proposed, so I think a rename isn’t out of the question if there’s something else everyone comes to like.
- Jon
Yeah, I like that idea. But where to host it? Pulling it from bro.org means an additional point of failure, so maybe it just all gets shoved into a separate gist file that lives on github?
- Jon
Yes I can reorganize all latest ideas into an alternate design document. Or if you meant get started working on the client, the open question about what metadata will be required has the most impact on how the client works, but like I said, I think things might also work if I start w/ the assumption that plugins are not required to present any metadata.
- Jon
That’s all I meant. Thanks.
Here’s a revised/alternate design based on feedback so far:
https://www.bro.org/development/projects/cban3.html
Note that I put these at unique URLs and didn’t update in place since they’re different enough that, if someone tried to follow along from the start of the thread, I didn’t think it would make sense to them.
- Jon
Looks great to me. Unsorted thoughts/notes:
- Let's keep BroControl integration in mind at leat. I agree that a
standalone client makes most sense right now, but if there's
something we can do that will make it easier for BroControl later to
integrate, that's worth preparing for.
- One idea along those integration lines (BroControl and even
elsewhere): would be nice to structure the client so that the main
functionality resides in a Python module with a well-defined API.
The client itself would then become just a small command-line
frontend to that library.
- Where's that backend-side "out-of-band process" located? Is that
part of the client as well? Or maybe at least part of that Python
module? Would be nice to keep it easy to operate for CBAN-like
repositories that people maintain externally.
- having both "upgrade" vs "update" commands sounds confusing, I'm
sure I would constantly mix up the two. Suggest to rename one of
them.
- What's the policy for namespaces (both script-land and plugin-land),
and what's the relationship to the CBAN path? Should people use
their GitHub name as namespace for their module? On the one hand
that would be an easy way to avoid conflicts. On the other hand that
makes forking difficult.
- Originally I wanted to suggest allowing more than one plugin per git
repository but I really like the idea to leverage submodules, so I'm
skipping that suggestion. 
- How do updating a plugin works now? If the author just updates the
remote code, the submodules won't move, so is there an additional
step there?
- Using a gist for meta data sounds good. We should then also have a
nice permanent *.bro.org URL redirecting there so that we hide the
actual location a bit for easier relocation.
- Any thoughts on distributing precompiled plugins? It would be nice
if, say, I could just install the Mac version of plugin XYZ without
having to compile it locally. The authors could optionally offer
selected prebuilds for whatever platforms they want to support.
Probably more a a feature for CBAN v2, but would be wort keeping
mind how binaries would fit in.
- Another maybe-v2 idea: if a plugin listed its configuration options
in a well-defined manner, tools like BroControl could pick up on
that and offer a configuration UI without peopling having to write
script code.
- Terminology 1: agree that we should find a better name for CBAN.
- Terminology 2: Using "plugin" as the entity name for everything is
confusing I think, as right now I think most people understand it as
something that gets compiled; nobody calls a script a plugin (unless
it's a plugin for a specific script-framework; but that's even more
confusing then :). The best alternative names I can come up with are
"module" or "package", which are also overloaded already, but at
least more generic. Other suggestions?
- We could create a public mailing list for CBAN for all discussions
of individual plugins/modules, including in particular for decisions
to remove something. Would be good to keep this all open and
transparent.
- I'm offended that my plugin is just "nice", but Seth's is *cool*!
Robin
- Terminology 1: agree that we should find a better name for CBAN.
BroForge?
- Terminology 2: Using "plugin" as the entity name for everything is
confusing I think, as right now I think most people understand it as
something that gets compiled; nobody calls a script a plugin (unless
it's a plugin for a specific script-framework; but that's even more
confusing then :). The best alternative names I can come up with are
"module" or "package", which are also overloaded already, but at
least more generic. Other suggestions?
But we are wrapping everything in the Bro plugin architecture, right? I guess I I don’t think of plugins as necessarily binary, but maybe we shift nomenclature completely and start calling plugins packages across our documentation. But if we refer to the same thing as a plugin sometimes and a package other times, I think it gets even more confusing.
- We could create a public mailing list for CBAN for all discussions
of individual plugins/modules, including in particular for decisions
to remove something. Would be good to keep this all open and
transparent.
I like the idea of a CBAN list.
CBAN is a good name. It’s short, easy to say, and what the acronym means, the “Comprehensive Bro Archive Network”, makes sense. But I don’t really have a dog in this fight.
- Let's keep BroControl integration in mind at leat. I agree that a
standalone client makes most sense right now, but if there's
something we can do that will make it easier for BroControl later to
integrate, that's worth preparing for.- One idea along those integration lines (BroControl and even
elsewhere): would be nice to structure the client so that the main
functionality resides in a Python module with a well-defined API.
The client itself would then become just a small command-line
frontend to that library.
Ack.
- Where's that backend-side "out-of-band process" located? Is that
part of the client as well? Or maybe at least part of that Python
module? Would be nice to keep it easy to operate for CBAN-like
repositories that people maintain externally.
I imagined it being part of the nightly process that does quality/metric gathering.
The code to do it could be part of the client, but I expect the aggregation process could be time consuming enough that a user wouldn’t be happy w/ it. Especially if CBAN grows to contain many submodules, I don’t think it would scale well for each client to have to do the metadata aggregation themselves.
Is it an important use case for the client to be able to interact w/ other repos that are structured like the one the Bro Team will be hosting? Seems less likely that people will want to do that especially if the CBAN repository is easy enough for people to submit to and the client can handle the case of adding individual external repos just fine.
The metadata from individual, externally added repos would be searchable along with the bulk/aggregated metadata. e.g. all the submodules a user has locally installed have the latest metadata available, but all the submodules the user has not installed will fall back on the aggregated metadata that was fetched the last time they ran an “update” command.
- having both "upgrade" vs "update" commands sounds confusing, I'm
sure I would constantly mix up the two. Suggest to rename one of
them.
I do confuse them too, but it’s also what some other package managers use (e.g. home-brew and apt-get). I’ll look at others package managers for more examples to see if I find something.
- What's the policy for namespaces (both script-land and plugin-land),
and what's the relationship to the CBAN path? Should people use
their GitHub name as namespace for their module? On the one hand
that would be an easy way to avoid conflicts. On the other hand that
makes forking difficult.
Didn’t really have plans/ideas for a namespace policy, but checking for conflicts could be added to the list of things the automated nightly quality/metrics thing checks for.
But maybe there’s a slicker thing you could add to the Bro language itself to let users specify a custom namespace mapping for certain file paths allowing them to resolve conflicts themselves.
- How do updating a plugin works now? If the author just updates the
remote code, the submodules won't move, so is there an additional
step there?
The client's “update” command will do a “git pull” in the parent repo and then a “git fetch” in any installed submodules. Any submodules that are not installed are a no-op because they are just pointers that got updated by the “git pull”. This means the client should now be aware of updates to all the plugins they have installed, but are still using the old version until they choose to perform the “upgrade” command which then moves the installed submodules to the latest version.
Or at least it will look something like that. I haven’t had a chance to test if there will need to be a separate “working copy” of submodules the user chooses to install.
- Any thoughts on distributing precompiled plugins? It would be nice
if, say, I could just install the Mac version of plugin XYZ without
having to compile it locally. The authors could optionally offer
selected prebuilds for whatever platforms they want to support.
Probably more a a feature for CBAN v2, but would be wort keeping
mind how binaries would fit in.
Didn’t think much about it yet, but I expect a plugin to be able to define its own build steps. For precompiled plugins they could likely just do a no-op for the build step. Maybe there’s other install paths that need to get set up that I’m forgetting, but that shouldn't be too much extra to get working. Let me know if you had other thoughts that would make this more complicated than I’m expecting.
- Terminology 2: Using "plugin" as the entity name for everything is
confusing I think, as right now I think most people understand it as
something that gets compiled; nobody calls a script a plugin (unless
it's a plugin for a specific script-framework; but that's even more
confusing then :). The best alternative names I can come up with are
"module" or "package", which are also overloaded already, but at
least more generic. Other suggestions?
Don’t have other suggestions, but “package” seems fitting because the concept of “package management” is an established term that people are familiar with.
- I'm offended that my plugin is just "nice", but Seth's is *cool*!
Maybe some people prefer “nice” to “cool”. BTW, originally you had just “some-plugin”, but I changed it.
- Jon
- having both "upgrade" vs "update" commands sounds confusing, I'm
sure I would constantly mix up the two. Suggest to rename one of
them.
I think this comes from Homebrew (and maybe other package managers as
well). I kinda got used to it in this context, but occasionally still
trip over it. So yeah, I also think we should use either "update" or
"upgrade," but not both.
- What's the policy for namespaces (both script-land and plugin-land),
and what's the relationship to the CBAN path? Should people use
their GitHub name as namespace for their module? On the one hand
that would be an easy way to avoid conflicts. On the other hand that
makes forking difficult.
Is it necessary to enforce this? Intuitively, I would just leave
namespacing to the user. Clearly, to be interoperable, a convention
could be using your github handle, but I don't see it being a necessity.
If multiple devs want to collaborate and enhance the same namespace, we
should not make this a cumbersome due to namespace hassles.
- Originally I wanted to suggest allowing more than one plugin per git
repository but I really like the idea to leverage submodules, so I'm
skipping that suggestion.
How do submodules scale? Or asked differently, what are the number
(order of magnitude) of submodules we're expecting?
- Using a gist for meta data sounds good. We should then also have a
nice permanent *.bro.org URL redirecting there so that we hide the
actual location a bit for easier relocation.
Is the idea that all meta data will reside in a single file? What format
would that file be, YAML? If each user repo has 20 lines of meta data,
then the file would be 2KB to track 100 repositories. Seems fine to me,
but I wonder how much will end up in there to get a tractable upper
bound.
- Any thoughts on distributing precompiled plugins? It would be nice
if, say, I could just install the Mac version of plugin XYZ without
having to compile it locally. The authors could optionally offer
selected prebuilds for whatever platforms they want to support.
Probably more a a feature for CBAN v2, but would be wort keeping
mind how binaries would fit in.
Binaries are a big project, I believe. Not only do we need to support
differently platforms, but also different OS versions and distributions.
It would probably mean we have to invest into a bigger Jenkins setup,
and then push the binaries into a CDN after a successful build.
- Terminology 1: agree that we should find a better name for CBAN.
Here are some bro'ish suggestions (not all are serious ones):
- brow (part of the logo already)
- broom (sweep new code into bro)
- broil (Bake your enhancements into bro)
- broad (extends Bro in a broader sense)
- broem (pleasing in a literary manner)
- bropane (hot stuff for your bro)
- brorito (original scripts from Mexico!)
- brotein (makes you grow like a bro)
- hebro (how about catering to Isreal?)
- bronx (or to NYC?)
And some random word forging:
- bkg (instead of FreeBSD's pkg)
- lion (animal I like)
- crank (crank up that Bro)
- scrit (tweak on "script")
- berk (a Bro perk...or short for Berkeley)
- bip (pip for Bro)
- bang (starts with a "b" and sounds hip)
- beco (Bro ecosystem)
The best alternative names I can come up with are "module" or
"package", which are also overloaded already, but at least more
generic.
Of the two, I prefer "module" because "package" to me reminds me of a
compiled thing, which is not always the case. Other than that,
"extension" comes to mind, or maybe "bundle."
- We could create a public mailing list for CBAN for all discussions
of individual plugins/modules, including in particular for decisions
to remove something. Would be good to keep this all open and
transparent.
...and a Gitter channel, once we have converged on a name :-).
Matthias
Here are some bro'ish suggestions (not all are serious ones):
- brow (part of the logo already)
- broom (sweep new code into bro)
- broil (Bake your enhancements into bro)
- broad (extends Bro in a broader sense)
- broem (pleasing in a literary manner)
- bropane (hot stuff for your bro)
- brorito (original scripts from Mexico!)
- brotein (makes you grow like a bro)
- hebro (how about catering to Isreal?)
- bronx (or to NYC?)And some random word forging:
- bkg (instead of FreeBSD's pkg)
- lion (animal I like)
- crank (crank up that Bro)
- scrit (tweak on "script")
- berk (a Bro perk...or short for Berkeley)
- bip (pip for Bro)
- bang (starts with a "b" and sounds hip)
- beco (Bro ecosystem)
I like bkg, broil and brow. What about bob (helps you building up you
Bro ;)?
Or just bro-ports
Is it necessary to enforce this? Intuitively, I would just leave
namespacing to the user.
No need to enforce, but it would be good to have some guidelines at
least. And part of the guidelines should be keeping the "Bro"
namespace reserved.
"extension" comes to mind, or maybe "bundle."
I like "bundle".
Robin
I think we'll need to play with that a bit still to see how exactly a
minimal script module would be distributed. The binary plugins require
a bit more structure, and are loaded a bit differently, than something
that has just scripts would need. Maybe we can simplify the plugin
layout a bit to support that case better, not sure. That said, even if
using all the same structure, in my head I still associate "plugin"
with one of the binary extensions (log writer, packet source) etc. and
that's also how the documentation uses that term currently.
Robin
I imagined it being part of the nightly process that does quality/metric gathering.
Yeah, makes sense.
Is it an important use case for the client to be able to interact w/
other repos that are structured like the one the Bro Team will be
hosting? Seems less likely that people will want to do that
especially if the CBAN repository is easy enough for people to submit
Yeah, agree it's less important with that liberal model. I would still
like to support it though unless it's a major pain.
But maybe there’s a slicker thing you could add to the Bro language
itself to let users specify a custom namespace mapping for certain
file paths allowing them to resolve conflicts themselves.
Conflict reporting sounds good, renaming could get confusing.
The client's “update” command will do a “git pull” in the parent repo
and then a “git fetch” in any installed submodules.
So does that mean the client ignores the version that the central CBAN
parent repository points to? Say Seth finished version 1.0 of his
cool-plugin and files a merge request. We add the submodule to CBAN;
it will point to 1.0. Then Seth updates to 1.1 on his end. CBAN would
still point to 1.0, but clients would just move their local clones to
1.1, ignoring the parent repository's state. Is that what you have in
mind?
Robin
How do submodules scale? Or asked differently, what are the number
(order of magnitude) of submodules we're expecting?
I imagine it will scale because when a user clones the main/parent CBAN repo, we’re telling them to just clone that and *not* recursively clone all the submodules (unlike the process for the ‘bro’ repo that you’re familiar with). Then the user will pick and choose the individual submodules they want. I don’t know much the average user will end up installing, but I’m guessing in the order of 10s and not 100s.
Does that make sense or is there something more to your concern I might be missing or misunderstood?
Is the idea that all meta data will reside in a single file? What format
would that file be, YAML? If each user repo has 20 lines of meta data,
then the file would be 2KB to track 100 repositories. Seems fine to me,
but I wonder how much will end up in there to get a tractable upper
bound.
I didn’t have a standardized format in mind to begin with, I just planned to do “the simplest thing that works” first and then let it evolve from there (clients can be updated in a way that’s resilient to changes in the format), but if enough people want to start w/ a particular format to begin with that’s fine too.
Binaries are a big project, I believe. Not only do we need to support
differently platforms, but also different OS versions and distributions.
It would probably mean we have to invest into a bigger Jenkins setup,
and then push the binaries into a CDN after a successful build.
I imagined it would still be left up to individual authors to make precompiled binaries available and support/maintain them. It would also be optional for them to make binaries available, not required.
- Jon
Is it an important use case for the client to be able to interact w/
other repos that are structured like the one the Bro Team will be
hosting? Seems less likely that people will want to do that
especially if the CBAN repository is easy enough for people to submitYeah, agree it's less important with that liberal model. I would still
like to support it though unless it's a major pain.
Ok, I’ll plan from the beginning to be able point to multiple CBAN-like repository structures and then complain if I find out it’s complicated to support that.
The client's “update” command will do a “git pull” in the parent repo
and then a “git fetch” in any installed submodules.So does that mean the client ignores the version that the central CBAN
parent repository points to? Say Seth finished version 1.0 of his
cool-plugin and files a merge request. We add the submodule to CBAN;
it will point to 1.0. Then Seth updates to 1.1 on his end. CBAN would
still point to 1.0, but clients would just move their local clones to
1.1, ignoring the parent repository's state. Is that what you have in
mind?
Yeah, the parent repo would always point to the first version that was submitted, but when a user chooses to install something, that submodule gets initialized/cloned and the user can choose to use whatever version of it they want.
And it won’t be the common case for a user to be doing any actual work inside the parent repo (unless they’re working on the cban client code), so it shouldn’t be a big deal if things like `git status` in the parent repo show a bunch of the “new commits” messages for each installed submodule. But I’ll also probably structure it so all submodules get cloned in to a subdirectory and then list that subdirectory in .gitignore to limit the noise.
- Jon