Broctl installation process clean up

Development
1

As part of the CMake porting process, I think it makes sense to clean up Broctl's initial installation process, which currently creates a dependence between some of the `broctl install` functionality and the Bro distribution source dir. That's not going to be compatible when we want to have Broctl packaged individually as a Bro optional/add-on package.

I propose that we let CMake take care of the dependency checking (other Bro components need to be installed first) and initial installation of the Broctl files while `broctl install` would still be responsible for generating scripts from templates, updating to new version of .cfg and site policy files, and syncing to worker nodes (at least I think that's a summary of what broctl does).

The specific functionality that would be lost is `broctl install --make` (used by current Makefile) and also the DevMode=1 feature that seems to basically turn `broctl install` into a `broctl install --make` in order to update the files from the distribution source. Instead, the `make install` target generated by CMake can accomplish that task and probably remind the user that they must also `broctl install` if they want to update scripts/config or sync to workers.

I think I've nearly thought this all the way through and wrote the details down here, for the curious:

https://wiki.ncsa.illinois.edu/display/BROWATERS/CMake+Porting+Progress#CMakePortingProgress-BroctlTasks%2FNotes

Good plan or bad plan?

- Jon

2

Log In - NCSA Wiki

Very nice, good job! That all sounds good. A few more comments
below.

As part of the CMake porting process, I think it makes sense to
clean up Broctl's initial installation process,

Yes, definitly.

I propose that we let CMake take care of the dependency checking
(other Bro components need to be installed first)

I agree. One reason for the current installation is to essentially
get around the fact that there's no coherent way of installing all
the pieces easily, and broctl's install tries to hide that as good
as it can and just assembles what it needs.

and initial installation of the Broctl files while `broctl install`
would still be responsible for generating scripts from templates,
updating to new version of .cfg and site policy files, and syncing
to worker nodes (at least I think that's a summary of what broctl
does).

That sounds good. The one additional thing we should think about
(per earlier mails) is getting rid of generating scripts from
templates. How much work do you think would it be to adapt the
installation to (1) generate a single shell script, say
broctl-config.sh[1], with all (relevant) configure options; and (2)
use static versions for all the helper scripts which then source
broctl-config.sh before doing anything else. That way, we should be
able to get rid of (nearly?) all the Target/TargetDev entries in
BroControl/install.py, leaving them to CMake instead. I'm thinking
we need to do that anyway eventually, and doing so later would then
again require to work on the installation.

[1] Ideally, this would be generated on a per-system basis, so that
nodes can be configured differently. However, that might turn out to
be a bit tricky and I'm fine leaving that out for now.

The specific functionality that would be lost is `broctl install
--make` (used by current Makefile)

No problem.

and also the DevMode=1 feature that seems to basically turn `broctl
install` into a `broctl install --make` in order to update the
files from the distribution source.

That's ready to go anyway, it's mainly a relict from broctl's first
version.

Good plan or bad plan?

Excellent plan!

The policy files into $broctl_prefix/share/bro ?

Better into a subdirectory of that, to keep them seperate from
Bro's.

One more thing: some easy way to change the default paths where the
various broctl pieces go would be nice; seems distributions will
need that (also, if it makes sense to change any of the default
paths generally (for broctl, or also Bro itself), now would be the
right time for that as well).

Robin

3

I agree. One reason for the current installation is to essentially
get around the fact that there's no coherent way of installing all
the pieces easily, and broctl's install tries to hide that as good
as it can and just assembles what it needs.

While I was looking into it, I was thinking that the Python distutils could have done the broctl installation. Not saying that I want to go with distutils here, just wondering if you had considered it since (from broccoli bindings thread) it sounds like it may be desirable for Bro to maintain distutils build/installation of Python stuff as a standard.

> and initial installation of the Broctl files while `broctl install`
> would still be responsible for generating scripts from templates,
> updating to new version of .cfg and site policy files, and syncing
> to worker nodes (at least I think that's a summary of what broctl
> does).

That sounds good. The one additional thing we should think about
(per earlier mails) is getting rid of generating scripts from
templates. How much work do you think would it be to adapt the
installation to (1) generate a single shell script, say
broctl-config.sh[1], with all (relevant) configure options; and (2)
use static versions for all the helper scripts which then source
broctl-config.sh before doing anything else. That way, we should be
able to get rid of (nearly?) all the Target/TargetDev entries in
BroControl/install.py, leaving them to CMake instead. I'm thinking
we need to do that anyway eventually, and doing so later would then
again require to work on the installation.

That doesn't sound hard to do, but I'm not sure I understand completely (and don't know what earlier mails you're referencing).

Do you mean that CMake installs something like a broctl-config.sh.in and that `broctl install` transforms it into the broctl-config.sh that the now-static scripts will source?

Or that everything is now static, broctl-config.sh and also the scripts?

> The policy files into $broctl_prefix/share/bro ?

Better into a subdirectory of that, to keep them seperate from
Bro's.

Right; thanks. I see that they get put in $prefix/share/bro/broctl currently.

One more thing: some easy way to change the default paths where the
various broctl pieces go would be nice; seems distributions will
need that (also, if it makes sense to change any of the default
paths generally (for broctl, or also Bro itself), now would be the
right time for that as well).

I'll try to anticipate as much as I can, but getting things perfect for packaging purposes will probably require another iteration after the basic CMake build is done (or else my head might explode).

- Jon

4

I want to go with distutils here, just wondering if you had
considered it

Initially I didn't do that as I didn't seen an easy way to get all
the special stuff the install used to do integrated with distutils
(perhaps I'm just not sufficiently familiar with it). Also, the
installation setup grew kind of organically. But I wouldn't be
opposed to switching to distutils, in fact I'd prefer it but don't
know much work that is and whether it's worth it. What do you think?

That doesn't sound hard to do, but I'm not sure I understand
completely (and don't know what earlier mails you're referencing).

(I think I discussed this briefly in earlier threads at least with
Seth, but perhaps that wasn't on the mailing list; not sure).

Do you mean that CMake installs something like a broctl-config.sh.in
and that `broctl install` transforms it into the broctl-config.sh
that the now-static scripts will source?

Yes, this is what I mean. There'd be only one dynamic file now,
broctl-config.sh generated by "broctl install", and all the rest is
static (much of the content of broctl-config.sh could be determined
at CMake-time already, however there's some stuff in there that
depends on broctl.cfg).

I'll try to anticipate as much as I can, but getting things perfect
for packaging purposes will probably require another iteration after
the basic CMake build is done (or else my head might explode).

Yes, I see that. Just wanted to throw it out there. :slight_smile:

Robin

5

> I want to go with distutils here, just wondering if you had
> considered it

I wouldn't be opposed to switching to distutils, in fact I'd prefer it but don't
know much work that is and whether it's worth it. What do you think?

You snipped my quote to make me say "I want to go with distutils here", but actually I'd rather just use CMake for now (mostly for consistency & comfort reasons). But a setup.py might still be considered later if it's more standard for packagers to use.

There'd be only one dynamic file now,
broctl-config.sh generated by "broctl install", and all the rest is
static (much of the content of broctl-config.sh could be determined
at CMake-time already, however there's some stuff in there that
depends on broctl.cfg).

Ok, will do that refactor along with the other clean up.

- Jon

6

You snipped my quote to make me say "I want to go with distutils here"

(... that was accidental, I didn't read it that way)

Ok, will do that refactor along with the other clean up.

Ok, thanks.

Robin