Beta schedule

Sounds like everybody feels we're getting ready to do our beta
release. Let's aim for early next week. Here's what I think is left to
do:

    - #511: Cleanup distribution a bit more (Robin)
    - #601: Write docs for notice framework (Seth)
    - #601: Cleanup and organize the documentation we have (Robin[1])
    - #544: Get scan.bro into shape (Seth)
    - #622: Some of these seem to addressed, others not. (John, can
            you take the lead closing this ticket? Some might just need a
            decision/reminder if what's we're currenty doing is the right
            thing.)

I'm fine pushing the rest to the final release. Let me know if anybody
see any other showstopper.

Robin

[1] I'll make sure things get online and make a pass over the content
if time permits. Anybody else, please also look through and edit what
you find.

Sounds great to me.

  .Seth

Maybe I missed them but what's with (all from #601):

  * Upgrade guide
  * How-to for new logging framework (how to filter, log file format,
    maybe how to add columns to the log file (i.e., how to use
    c$PROTOCOL))
  * Complete script reference (though that can wait to final release)

cu
gregor

Maybe I missed them but what's with (all from #601):

* Upgrade guide

tracked in more detail here: http://tracker.bro-ids.org/bro/ticket/510

and put up on the site here: http://www.bro-ids.org/documentation/upgrade.html

* How-to for new logging framework (how to filter, log file format,
   maybe how to add columns to the log file (i.e., how to use
   c$PROTOCOL))

Think that's mostly done: http://www.bro-ids.org/development/logging.html

* Complete script reference (though that can wait to final release)

Does that mean the generated bro script docs (think I'm going to start calling them "Broxygen docs" for short) ?

If so, I didn't look at how far Robin got on the www new-git branch, but part of that will help incorporate those docs into the website.

Another general task might be to cleanup the scattered TODO/XXX's.

- Jon

Updates:

- I have spent some time redoing how git files are pulled in by
  www.bro-ids.org. That now allows us to have separate sets of docs
  for release, beta, and master; and generally makes usage easier.

- I have also cleaned up the submodules to unify README structure,
  licenses, etc.

- I'm making releases (with tarballs) for all the submodules, except
  broctl. There shouldn't be changing much for them during the Bro
  beta period, and even if, we can always release new versions. Jus
  trying to get them out of the way.

I'm planing to push all this out tomorrow.

Robin

- I'm making releases (with tarballs) for all the submodules.

I'm trying to build tar-files for all our submodules but I'm running
into a number of things.

First, afaics, not all modules yet provide the capability to create
the tarballs (or at least I can't directly figure out how).
Specifically:

    - capstats, binpac, and bro-aux: they have "dist" Makefile
      targets, but running thise gives me:

        make[1]: *** No rule to make target `package_source'. Stop.

    - broccoli-ruby: No "dist" Makefile target and I have no idea how
    to do that for Ruby packages (I've just added an empty dummy one.)

Then, the top-level Bro "make-src-packages" also builts the Broccoli
and BroControl tgz. I think it would be better to leave that to the
submodules themselves, otherwise things get inconsistent (because we
have all these further submodules as well, which do it themselves).
So I've removed those lines from make-src-packages.

I also noticed that Bro's make-src-packages essentially just tars
everything that's in the source directory (except for some tmp files
etc.). There's a problem with that: I'm sure that sometimes I will
have additional files lying around there which will then end up in the
distribution. Is there a way to make sure that only stuff under
version control gets actually into the tgz? (And please remind me:
what was the reason that we can't just let CMake do the tarballs?)

Here's how I'd like to do the tarballs: I've written a script that
recursively goes through all submodules and runs "make dist" there.
It's in aux/devel-tools now. The script then collects all the
resulting tarballs at the top-level with the right naming/path scheme
for copying them over to the web server. For that to work, we need to
ensure that (1) each Makefile has indeed a working "dist" target, and
(2) that that targets outputs the path to the generated tarball so
that the top-level script can pick it up. Most CMake-builds do (2)
already and for the Python modules I've added output in the form of
"Package: relative/path/to/tgz" (which is then grepped for).[1]

Robin

[1] It works to write out multiple lines like that if more than one
tgz is built (like bro vs bro-all). Btw, I'm wondering if we should
name them the opposite way: bro-2.0.tgz would include all submodules,
and bro-2.0-minimal.tgz would not. Same scheme for BroControl and
Broccoli (where we don't do two tarballs yet).

- I'm making releases (with tarballs) for all the submodules.

I'm trying to build tar-files for all our submodules but I'm running
into a number of things.

First, afaics, not all modules yet provide the capability to create
the tarballs (or at least I can't directly figure out how).
Specifically:

   - capstats, binpac, and bro-aux: they have "dist" Makefile
     targets, but running thise gives me:

       make[1]: *** No rule to make target `package_source'. Stop.

   - broccoli-ruby: No "dist" Makefile target and I have no idea how
   to do that for Ruby packages (I've just added an empty dummy one.)

The package_source target is something provided by CMake, and adding it would require changing top-level CMakeLists.txt to include the cmake/ConfigurePackaging.cmake script, but I don't think that's ultimately the way to go...

Then, the top-level Bro "make-src-packages" also builts the Broccoli
and BroControl tgz. I think it would be better to leave that to the
submodules themselves, otherwise things get inconsistent (because we
have all these further submodules as well, which do it themselves).
So I've removed those lines from make-src-packages.

Yeah, sounds good -- repos for which we we want to create recursive packages should handle that themselves...

I also noticed that Bro's make-src-packages essentially just tars
everything that's in the source directory (except for some tmp files
etc.). There's a problem with that: I'm sure that sometimes I will
have additional files lying around there which will then end up in the
distribution. Is there a way to make sure that only stuff under
version control gets actually into the tgz?

I think we need to use a combination of `git ls-files` to generate a package manifest as input to `tar -T`.

The trick is that all non-recursive source packages need to include the contents of their cmake/ submodule if it exists.

For the recursive package bundles (Bro, BroControl, Broccoli), the output of a recursive `git ls-files` needs to be altered to prepend the right relative path of submodules.

(And please remind me:
what was the reason that we can't just let CMake do the tar balls?

So that the packager doesn't need to have CMake as a dependency.
And actually it's worse than that: the packaging configuration provided by CMake is also currently tied to the full suite of ./configure dependency checks as if one were going to actually build the project.
(This was originally pointed out as a pretty bad inconvenience by Craig Leres)

Here's how I'd like to do the tarballs: I've written a script that
recursively goes through all submodules and runs "make dist" there.
It's in aux/devel-tools now. The script then collects all the
resulting tarballs at the top-level with the right naming/path scheme
for copying them over to the web server. For that to work, we need to
ensure that (1) each Makefile has indeed a working "dist" target, and
(2) that that targets outputs the path to the generated tarball so
that the top-level script can pick it up. Most CMake-builds do (2)
already and for the Python modules I've added output in the form of
"Package: relative/path/to/tgz" (which is then grepped for).[1]

Robin

[1] It works to write out multiple lines like that if more than one
tgz is built (like bro vs bro-all). Btw, I'm wondering if we should
name them the opposite way: bro-2.0.tgz would include all submodules,
and bro-2.0-minimal.tgz would not. Same scheme for BroControl and
Broccoli (where we don't do two tarballs yet).

Sounds good, I'll work on cleaning things up.

- Jon

Yeah, sounds good -- repos for which we we want to create recursive
packages should handle that themselves...

Right.

I think we need to use a combination of `git ls-files` to generate a package manifest as input to `tar -T`.

I just remembered "git clean". That could delete everything we don't
want?

So that the packager doesn't need to have CMake as a dependency. And
actually it's worse than that: the packaging configuration provided by
CMake is also currently tied to the full suite of ./configure
dependency checks as if one were going to actually build the project.
(This was originally pointed out as a pretty bad inconvenience by
Craig Leres)

Hmm, ok. Not sure I'd see that as a showstopper, but I don't recall
the full discussion right now.

Sounds good, I'll work on cleaning things up.

Ok, thanks!

Robin

We are getting really close!

    - #511: Cleanup distribution a bit more (Robin)
    - #601: Cleanup and organize the documentation we have (Robin[1])

Done with these, except that I'm still planing to make a pass over the
upgrade docs.

    - #622: Some of these seem to addressed, others not. (John, can
            you take the lead closing this ticket? Some might just need a
            decision/reminder if what's we're currenty doing is the right
            thing.)

Closed.

    - #601: Write docs for notice framework (Seth)
    - #544: Get scan.bro into shape (Seth)

I think Seth is working on these.

Seth, one more: Almost all tests pass for me now (just updated a few
more baselines). The one exception is bro-testing:tests.m57-long,
which still gives me a long list of diffs. That's waiting for you I'm
afraid ...

Once these are done, I'll be happy to push out the beta.

I just remembered "git clean". That could delete everything we don't
want?

Added that part to the `make-release` script.

And all the `make dist` targets should all work better I think; try it out?

- Jon

Thanks, looks good, but one more request:

Added that part to the `make-release` script.

The disadadvantage of this is that is erases all local changes right
inside the repository I'm working in. Could "make dist" instead run
the "git clean" inside the temporary copy that it's creating? (Which
would probably mean that it needs to copy .git over first, then run
git clean, and then rm .git). That way nothing I have lying around
locally would be touched. I'm sure otherwise I will at some point
accidentally delete something.

Also, the "git submodule foreach ..." descends into all subdirectories
but doesn't execute inside the top-levle parent module. I think we
need another "clean ..." at the top-level as well, right?

Robin

The disadadvantage of this is that is erases all local changes right
inside the repository I'm working in. Could "make dist" instead run
the "git clean" inside the temporary copy that it's creating? (Which
would probably mean that it needs to copy .git over first, then run
git clean, and then rm .git). That way nothing I have lying around
locally would be touched. I'm sure otherwise I will at some point
accidentally delete something.

Ok, I'll change that.

Also, the "git submodule foreach ..." descends into all subdirectories
but doesn't execute inside the top-levle parent module. I think we
need another "clean ..." at the top-level as well, right?

yeah, forgot about that, but I'll remove that entirely since it will be done in the `make dists`

- Jon

It just happened ... Typing "make distclean" is deleting the repos in
testing/external/*. So let's please switch that back to the previous
"rm -rf build".

Robin

It just happened ... Typing "make distclean" is deleting the repos in
testing/external/*. So let's please switch that back to the previous
"rm -rf build".

Sorry. I will fix it.

- Jon