Tarballs

Development
1

Here's a proposal for tar-balls to offer for download in the future,
in terms of which repos are in there:

    bro-a.b.c.tgz
        bro.git
        binpac.git
        bro-aux.git

    broccoli-d.e.f.tgz
        broccoli.git
        broccoli-python.git

    broctl-g.h.i.tgz
        broctl.git
        capstats.git
        trace-summary.git
        pysubnettree.git
        broccoli.git
        broccoli-python.git

    bro-scripts-<timestamp?>.tgz [1]
        bro-scripts.git

    bro-all-a.b.c.tgz (using Bro's version number).
        All repos (except bro-scripts.git I guess)

When untar'ing, they'd all extract into the directories the super
module lays out for them (which means it doesn't matter whether one
gets broccoli-x.y.z.tgz on top of broctl-x.y.z.tgz).

We could do our binary packages along a similar structure.[2

Thoughts?

Robin

[1] I see the main way to distribute these as indvidually via the
web server. However, seems there's nothign wrong with also offering
them all at once for download.

[2] Distributions will want it separeately per repo, but for, e.g.,
an MacOS package, it seems more convinient to have few pieces to
download/install).

2

Here's a proposal for tar-balls to offer for download in the future,

...

Thoughts?

I think maybe the broctl source package wouldn't need to include broccoli since that's got it's own?

What you outlined seems to make the most sense to me in the cases where broccoli or broctl come out with new releases that are yet included in a bro release -- then a user could try to upgrade those modules (hopefully) without affecting bro.

Do you anticipate broccoli & broctl frequently having releases not in sync w/ bro? If not, I'm thinking it's easier to just have a bro-all and a bro-scripts source tarballs.

Was there other benefits associated with having those extra source tarballs (broccoli, broctl) that you had in mind?

We could do our binary packages along a similar structure.[2]

[2] Distributions will want it separeately per repo, but for, e.g.,
an MacOS package, it seems more convinient to have few pieces to
download/install).

Yeah, I think distributions will likely put finer grained packages in their repositories, but personally, if I were downloading a prebuilt package from the website, I'd rather just download a single RPM, DEB, OS X installer, etc, that has everything I should ever need.

- Jon

3

I think maybe the broctl source package wouldn't need to include
broccoli since that's got it's own?

Yeah, sounds right.

Do you anticipate broccoli & broctl frequently having releases not
in sync w/ bro?

For broctl, yes, that I expect regular independent releases. As
broctl sits mostly on top of the core Bro, it can be extended
without touching Bro itself and there are number of things for
broctl on our todo list.

For broccoli, the reason is a different one: it's something that can
be installed on systems that are not running Bro themselves but
merely *talk* to a Bro system. For example, if some site wants to
have all their syslogs to forward to Bro, they'll need t install
Broccoli evyerywhere. Having a separate source distrubution for that
seems useful.

Yeah, I think distributions will likely put finer grained packages
in their repositories, but personally, if I were downloading a
prebuilt package from the website, I'd rather just download a single
RPM, DEB, OS X installer, etc, that has everything I should ever
need.

Same here, but I think the arguments above apply here as well: would
be nice to be able to update broctl separately, and to install
broccoli indepedently of anything else.

Robin

4

> I think maybe the broctl source package wouldn't need to include
> broccoli since that's got it's own?

Yeah, sounds right.

Alright. I think CMake/CPack can be made to behave and produce this package structure.

> I'd rather just download a single RPM, DEB, OS X installer, etc, that has
> everything I should ever need.

Same here, but I think the arguments above apply here as well: would
be nice to be able to update broctl separately, and to install
broccoli indepedently of anything else.

Sounds good; I'll agree that the binary packages can mimic the source ones.

Want to decide what formats of binary packages to provide?

I think I can claim that DEB, RPM, Mac PackageMaker, and tarball formats are possible by relying on CPack, but it looks a little ugly...

For generating RPMs, it's difficult to get full control of package metadata (description, license, maintainer name/email), so the package might not look so professional.

For generating DEBs, the package dependencies will not be set automatically, so CPack is basically unusable. As a workaround, "alien" can be used to generate a DEB from RPM w/ correct dependencies, but again it's difficult to get control of metadata.

For Mac PackageMaker or tarball formats, one deficiency I remember is that you can't unpack to just any directory and have bro work because the default policy search dirs are set at compile time. Also, with tarballs we'd probably have to link all the dependencies statically; with PackageMaker, vanilla OS X comes with all required dependencies and I think it's sufficient to just ship the optional ones with the package.

Given that description, how comfortable are you with relying on CPack to make RPMs (converted to DEBs via alien) and Mac PackageMaker binary packages?

5

I think I can claim that DEB, RPM, Mac PackageMaker, and tarball
formats are possible by relying on CPack, but it looks a little
ugly...

By tarballs, you mean tarballs of binaries, right? I'm actually not
sure we need those. Let's skip if nobody objects.

Let's do just generic RPMs and Mac PackageMaker (and source tarballs
of course). No problem if the RPMs don't look perfect as long as
they *work* fine at least on a standard RedHat system.

In addition, let's then work with others to tailor to distributions
(and rely on their judgement in terms of what the best solution is
build those packages):

    - Justin for DEB.

    - "Real" RPMs whenever we find somebody willing to do them for
      some distribution.

    - Craig Leres, of LBL, for FreeBSD ports. He has offered to help
      with that when I talked to him last week. I'll get you in
      touch with him when you think you are ready for that.

For Mac PackageMaker or tarball formats, one deficiency I remember
is that you can't unpack to just any directory and have bro work
because the default policy search dirs are set at compile time.

Shouldn't be a problem for the Mac packages as they will install
into a predefined location, won't they?

with PackageMaker, vanilla OS X comes with all required dependencies
and I think it's sufficient to just ship the optional ones with the
package.

Sounds good.

Robin

6

I think what was meant is source tarballs as opposed to getting source through git.

7

Oh, ok, but then I don't understand the point about static linking.
Why would that matter for a source distribution? I must be missing
something.

Robin

8

I could be the confused one. I'll let Jon clarify, but I agree binary tarballs aren't needed.

9

By tarballs, you mean tarballs of binaries, right? I'm actually not
sure we need those. Let's skip if nobody objects.

Right; I agree to skip.

Let's do just generic RPMs and Mac PackageMaker (and source tarballs
of course). No problem if the RPMs don't look perfect as long as
they *work* fine at least on a standard RedHat system.

Ok.

> For Mac PackageMaker or tarball formats, one deficiency I remember
> is that you can't unpack to just any directory and have bro work
> because the default policy search dirs are set at compile time.

Shouldn't be a problem for the Mac packages as they will install
into a predefined location, won't they?

The user gets a choice if they have more than one volume.

> with PackageMaker, vanilla OS X comes with all required dependencies
> and I think it's sufficient to just ship the optional ones with the
> package.

Sounds good.

Instead of shipping optional libs w/ Mac package, this changed to linking statically for optional libs (libmagic, libgeoip, google perftools) as per other thread.

- Jon

10

I could be the confused one. I'll let Jon clarify, but I agree binary
tarballs aren't needed.

I did mean binary tarballs. Sorry for the confusion; I just added them in for completeness.

- Jon

11

Good point. We could:

- move bro to, say, bro.bin, and do a wrapper script bro
  that presets BROPATH.

- simply tell the user that it's configured for a standard location
  and the user needs to set BROPATH if he wants something else.

Just the latter is probably fine if we can point that out easily
somehow.

Robin

12

- simply tell the user that it's configured for a standard location
and the user needs to set BROPATH if he wants something else.

Just the latter is probably fine if we can point that out easily
somehow.

Ok, pointing it out on the website as well as in a specific README that gets displayed as part of the PackageMaker install process should probably cover it.

- Jon

13

For generating RPMs, it's difficult to get full control of package metadata (description, license, maintainer name/email), so the package might not look so professional.

For generating DEBs, the package dependencies will not be set automatically, so CPack is basically unusable. As a workaround, "alien" can be used to generate a DEB from RPM w/ correct dependencies, but again it's difficult to get control of metadata.

I would think that once we have the meta-data framework for deb and rpm,
we won't need to really touch them, right. So supporting deb and rpm
even without CPack seems to be a do-once task, right?
Or am I missing something?

cu
Gregor

14

I would think that once we have the meta-data framework for deb and rpm,
we won't need to really touch them, right. So supporting deb and rpm
even without CPack seems to be a do-once task, right?
Or am I missing something?

I think that's right, but don't quote me. Having CPack generate packages is just theoretically easier and less overall work (although that's debatable because it's poorly documented and hard to figure out sometimes) if you're not worried about fine-tuning things.

So since it seems we want to delegate the worry of fine-tuning to per-platform maintainers and I already went through the pain of figuring out CPack, we might just go ahead and use it to create packages for our own purposes.

Even if we went outside CPack to construct a meta-data framework, it's questionable how much of it would be re-usable by per-platform maintainers -- our binary packaging setup probably has too much clumped into each package (e.g. even broccoli python bindings might need to be split out into its own package, distinct from broccoli).

- Jon