Testing and Docs for Packages

Hi all,

building some small packages and playing around with a package for the
af_packet plugin (https://github.com/J-Gras/bro-af_packet-plugin), I
came across a question: How to deal with testing? For the
intel-extensions package (https://github.com/J-Gras/intel-extensions) I
adapted some scripts I found. While executing the tests is not an issue
(maybe only for me), they are hidden somewhere in the bro-pkg
directories, as of course only the scripts inside the script_dir are moved.

In general I think, making test cases available for users of a package
could be quite helpful. Further, I think we have also already mentioned
the possibility of compatibility checking regarding the installed Bro
version by executing tests. Thus I would propose introducing test_dir to
bro-pkg.meta for tests and a test command to execute the tests for
bro-pkg as a first step.

Another thing that would be great to have for packages would be
documentation. I've experimented with the broxygen feature but the doc
about generating docs is not that exhaustive :wink: I think providing an
easy-to-use mechanism to generate documentation for thrid-party scripts
would be great. Ideally the generated documentation would link to Bro's
online doc for built-in types and scripts.

What do you think about testing and docs for packages?

Jan

In general I think, making test cases available for users of a package
could be quite helpful. Further, I think we have also already mentioned
the possibility of compatibility checking regarding the installed Bro
version by executing tests. Thus I would propose introducing test_dir to
bro-pkg.meta for tests and a test command to execute the tests for
bro-pkg as a first step.

Yes, seems useful. I’d do it like:

1) Add `bro-pkg test <package>` command.
2) Add “test_command” field to bro-pkg.meta

The “test_command” is more general than “test_dir" — the command could just `cd test_dir` if needed and there’s no other reason bro-pkg needs to know the dir where tests are stored, is there?

Other questions:

1) Is it fine for `bro-pkg test <package>` to operate on the installed version of the package or are there expectations of testing a package in an isolated sandbox without installing it? I think the former is more useful since it may catch odd inter-package conflicts that wouldn’t show up when testing in isolation.

2) Should we put btest on PyPi, add it as a dependency to bro-pkg, and make it the canonical testing framework for packages? This gives devs a straightforward way to proceed w/ writing tests and guarantees that bro-pkg users always have the ability to run them.

(There is a “btest” package on PyPi, but not the one we know, so not sure how to resolve that...)

Another thing that would be great to have for packages would be
documentation. I've experimented with the broxygen feature but the doc
about generating docs is not that exhaustive :wink: I think providing an
easy-to-use mechanism to generate documentation for thrid-party scripts
would be great. Ideally the generated documentation would link to Bro's
online doc for built-in types and scripts.

If the problem is that there’s a lack of examples/templates for generating script API docs via broxygen or that it simply doesn’t work at the moment, then yes, that’s something to fix.

But regarding the direction of autogenerated package docs in general, maybe it makes sense to work on that in conjunction with a web-frontend for package sources (e.g. a package repository browser). A package would be able to generate its docs independent of that, but if the web-frontend is going to become a go-to place for looking at package info/stats/docs and it takes care of creating all that without package authors having to do anything, then that seems like the superior/common use-case.

- Jon

Yes, seems useful. I’d do it like:

1) Add `bro-pkg test <package>` command.
2) Add “test_command” field to bro-pkg.meta

The “test_command” is more general than “test_dir" — the command could just `cd test_dir` if needed and there’s no other reason bro-pkg needs to know the dir where tests are stored, is there?

Although I am not sure whether this degree of flexibility is really
needed, I would assume it doesn't hurt either. In any way, the user
should be informed that "something" will be executed and the user should
trust the packet author/source.

1) Is it fine for `bro-pkg test <package>` to operate on the installed version of the package or are there expectations of testing a package in an isolated sandbox without installing it? I think the former is more useful since it may catch odd inter-package conflicts that wouldn’t show up when testing in isolation.

I think testing on the installed version is fine. Installation might be
in particular necessary for packages containing plugins.

2) Should we put btest on PyPi, add it as a dependency to bro-pkg, and make it the canonical testing framework for packages? This gives devs a straightforward way to proceed w/ writing tests and guarantees that bro-pkg users always have the ability to run them.

Ha, I forgot that bro-pkg is published using PyPi. Adding btest as a
dependency sounds great to me.

If the problem is that there’s a lack of examples/templates for generating script API docs via broxygen or that it simply doesn’t work at the moment, then yes, that’s something to fix.

Looking at https://www.bro.org/development/howtos/autodoc.html, I wasn't
able to generate anything for my custom script. Looking into the Bro
code I could deduce the meaning of the broxygen.conf values and was able
to generate reST. I didn't try to generate HTML. A 3-step guide how to
generate doc for a custom script using the HTML template would be great.

But regarding the direction of autogenerated package docs in general, maybe it makes sense to work on that in conjunction with a web-frontend for package sources (e.g. a package repository browser).

Cool! I wasn't aware that a web-frontend is on the list. In that case,
any autogeneration of docs is indeed something to consider in this context.

Best regards,
Jan

Just to add my two cents to this, because automated testing is actually
one of the things that I really think package managers should do...

1) Add `bro-pkg test <package>` command.

Might it also make sense to just run the test on installation, before the
package is actually installed, to see if it works on the environment of
the user?

This might make it much easier for users (& developers) to identify early
when it is something wrong. And bro-pkg could just abort (or ask a user if
it should continue) if a test fails.

2) Add “test_command” field to bro-pkg.meta

The “test_command” is more general than “test_dir" — the command could just `cd test_dir` if needed and there’s no other reason bro-pkg needs to know the dir where tests are stored, is there?

Other questions:

1) Is it fine for `bro-pkg test <package>` to operate on the installed version of the package or are there expectations of testing a package in an isolated sandbox without installing it? I think the former is more useful since it may catch odd inter-package conflicts that wouldn’t show up when testing in isolation.

I actually think it would be neat to do this isolated, especially given
that this enables testing before installing. It also makes it easier to
create something like "smokers" (Bro installations that just tro tu run
all testsuites of all available packages with a newer version to see if
something went wrong).

Running on the installed version might be a neat bonus, but I actually see
the other as more interesting.

Johanna

1) Add `bro-pkg test <package>` command.

Might it also make sense to just run the test on installation, before the
package is actually installed, to see if it works on the environment of
the user?

Yes, I like that idea. (I’d also want a flag or config option to opt-out of that behavior).

I actually think it would be neat to do this isolated, especially given
that this enables testing before installing.

Not sure I follow. Can you explain further?

From a typical user perspective, I think they would care more that the package’s tests pass in the final, installed state and it plays nice with any other site-specific stuff they have going on. Aborting an installation on test failure is also still possible — instead of bro-pkg cleaning up an isolated sandbox, it does the standard ‘remove’ operation to delete installed files.

It also makes it easier to
create something like "smokers" (Bro installations that just tro tu run
all testsuites of all available packages with a newer version to see if
something went wrong).

Can you also go into more detail on what you’re thinking there?

If there's concerns about accidentally corrupting an existing/production bro installation, the alternative I’d suggest would be to set up a separate bro-pkg config file for the smoke tests that would have bro-pkg install stuff in an isolated location. This allows users to explicitly define the testing sandbox for themselves.

- Jon

I also think it would be quite useful to test packages before
installing them, that gives a chance to catch problems before changing
anything (including things like: missing/broken/wrong dependencies;
lack of something OS-specific the package needs (say, it's a
Linux-only plugin); generally things that are special about the local
Bro installation)

Robin

Maybe I’m not so much questioning whether to run tests before or after installation, but rather if the testing sandbox should include everything from the current installation in addition to the new/updated packages. i.e. the pre-installation testing sandbox could be initialized with a copy of current installation dirs.

I would think the user’s expectation when running tests is to answer the question “will this bro-pkg operation break my setup?” You can’t answer that without testing within the context of their current installation environment. The reason for that is that packages have much potential to change core bro functionality and interfere with each other’s operation.

But for that same reason, it may also make it much harder for people to write unit tests for their package that are precise enough to not cause harmless failures in the presence of other packages — e.g. you couldn’t just check a baseline http.log as some other installed package could have altered it by adding a field, etc.

Summary of approaches/tradeoffs:

1) separate testing environment for each package
- worse at answering “will this bro-pkg operation break my setup?”
- easier to write stable tests

2) single testing environment for all packages
- better at answering “will this bro-pkg operation break my setup?” *if* package tests are written well
- harder to write stable tests

Neither seems great. I guess I plan to do (1) since it is easier on package authors and less likely to waste users time looking into harmless test failures (ones due to tests that are written too broadly).

Let me know if there’s other ideas/suggestions.

- Jon

Yeah, (1) makes sense most to me too. Otherwise the author of the
package, when writing tests, would have to shoot for a moving target
that he doesn't control. I think we have to accept that tests won't be
able to cover every possible Bro installation; they are a first line
of defense against making sure nothing fundamentally broken. Said
differently, the tests generally cannot answer the question "will this
bro-pkg operation break my setup"; what they answer is "is this
package ok?".

Robin

> I actually think it would be neat to do this isolated, especially given
> that this enables testing before installing.

Not sure I follow. Can you explain further?

Sorry - what I meant is that the tests can run before the packages are put
into the bro directory, so you can see if they will work with the
installed Bro version (or potentially system configuration) before putting
the files in. So you can use it as a prerequisite check for installation.

The other way round, you have to roll back after putting them already
there - unless I misunderstood something.

Plus - even in this case, shouldn't you be able to load the user scripts
by loading local.bro? Meaning we could even run all the tests twice - once
just with the default Bro installation, and once with the user changes,
both before installing the scripts, which could even give an indication if
Bro or other packages are at fault.

> It also makes it easier to create something like "smokers" (Bro
> installations that just tro tu run all testsuites of all available
> packages with a newer version to see if something went wrong).

Can you also go into more detail on what you’re thinking there?

If there's concerns about accidentally corrupting an existing/production
bro installation, the alternative I’d suggest would be to set up a
separate bro-pkg config file for the smoke tests that would have bro-pkg
install stuff in an isolated location. This allows users to explicitly
define the testing sandbox for themselves.

No, the idea would be more along the lines that, in this case, you might
actually never want to really install the package; you just want to see if
the tests can pass. Though, admittedly, this can once again be
accomplished by just immediately uninstalling afterwards.

Johanna