maybe migrating Zeek CI from Travis to Cirrus

Development
1

I experimented last week moving the CI config to Cirrus and found
several benefits to making that switch. Example build in my Zeek
fork:

https://cirrus-ci.com/build/6738081318699008

Notable differences from the Travis config:

* Free for open-source projects, can allocate up to 8 cpus (we get 2
cpus on Travis)

* Concurrent tasks up to 8 Linux, 2 FreeBSD, and 1 macOS (Travis is 8
total, doesn't offer FreeBSD, and we haven't configured it for macOS).
This is also limited per _user_ not per repo, so misc. PRs will no
longer backup the CI queue like they do on Travis. E.g. to quote
docs: "For example, if you have 10 active contributors to a repository
then you can end up with 130 tasks running in parallel for the
repository"

* I've set up pre-built Dockerfiles for all Linux systems to make it
faster to get those up and running rather than wait to download
dependencies every time (and potentially encounter transient network
issues as we see sometimes on Travis). This also makes it just as
easy as Travis to reproduce the CI environment locally (Travis also
used docker, but didn't organize things as standard Dockerfiles)

* I've set the external pcap files (~850mb) to get cached after the
first successful build (or auto-reuploaded if they ever change later)
and downloading from the cache takes around 1 minute rather than
sometimes 15-20 on Travis

* The above differences amount to getting CI results in about 12-30
minutes (macOS and memleak task are still a bit slow) rather than
50-60 minutes on Travis.

So my idea going forward (let me know if there's other thoughts):

I'll set up the main Zeek GitHub organization/repo to use Cirrus in
parallel with existing Travis CI. If it performs well for several
months, we can think about completely migrating off Travis. Don't
necessarily need to move other projects to it, but might anyway in
spare time. Also definitely would need to move the Coverity Scan cron
somewhere else. IMO, this doesn't fit in with the CI config, it just
needs to run regularly somewhere, so GitHub Actions may be a good fit
for that.

Other caveats:

* I haven't configured Cirrus to access the external private test
suite. IMO, we might just stop using that, since it's awkward for an
open-source project: we don't give access to private pcaps from
arbitrary PRs that trigger CI, but then what happens if the merger
finds failures in the private test suite (or forget to run it before
merging) ? Either need to give access to the private pcaps or else
the core team basically becomes responsible (and I can imagine some
new features/PRs being non-trivial to hunt down diffs ourselves).

* Haven't figured out how to configure email notifications on Cirrus
CI failures, but can figure that out once it's up and running.

- Jon

2

I’m pretty impressed by the build time improvement.

A few questions:

  1. Can we build arbitrary branches on Cirrus? To do that on Travis you either had to make a draft PR off the branch, or push a commit to your branch that whitelisted the branch name and then manually make a build with a cron job. Neither were great solutions.
  2. Why move the Coverity build? Does it not offer a cron-like interface for running regular builds?
  3. Can you move the travis-job script up into the ci directory instead of leaving it in testing?

Tim

3

1. Can we build arbitrary branches on Cirrus?

Yes, it's a bit awkward how you do it, but I guess better than Travis.
In Cirrus, you can manually trigger an arbitrary branch, but like
Travis, the default is to reject branches that are not whitelisted:

But in Cirrus, you also get the option to override the .cirrus.yml
directly in the web interface for manually triggering a branch. So
basically you can copy-paste the .cirrus.yml and remove the "only_if"
lines, no temporary hacks needed to be committed into the branch
itself.

2. Why move the Coverity build? Does it not offer a cron-like interface for running regular builds?

Didn't look like Cirrus has cron-like functionality yet, see

But also the Coverity build doesn't exactly fit the same mold as the
other tasks. If we didn't have any Coverity usage limitations (think
it's 1 build per day?) and it was required to run on every new commit
in `master`, then I'd say it would be ideal to have that ported into
Cirrus, too, but since that's not the case, I don't have much
preference where it runs except doesn't seem efficient to pay to
bypass the 50 minute Travis time limit anymore just for doing the
Coverity build if GitHub Actions has a free 6 hour time limit and also
supports cron workflows.

3. Can you move the travis-job script up into the ci directory instead of leaving it in testing?

Just to keep things a bit more organized while Travis is still being
used in parallel with Cirrus? Sure, but eventually travis-job can get
removed: rewrote/factored bits of that into the other ci/*.sh scripts
for Cirrus to use.

- Jon

4

1. Can we build arbitrary branches on Cirrus?

Yes, it's a bit awkward how you do it, but I guess better than Travis.
In Cirrus, you can manually trigger an arbitrary branch, but like
Travis, the default is to reject branches that are not whitelisted:

https://github.com/jsiwek/zeek/blob/94c4790610cf279157ecd536df144fadb83835c1/.cirrus.yml#L13-L16

But in Cirrus, you also get the option to override the .cirrus.yml
directly in the web interface for manually triggering a branch. So
basically you can copy-paste the .cirrus.yml and remove the "only_if"
lines, no temporary hacks needed to be committed into the branch
itself.

That sounds reasonable enough. It just always bothered me having to do weird tricks with the actual branch to make it build.

2. Why move the Coverity build? Does it not offer a cron-like interface for running regular builds?

Didn't look like Cirrus has cron-like functionality yet, see
Support cron builds · Issue #39 · cirruslabs/cirrus-ci-docs · GitHub

But also the Coverity build doesn't exactly fit the same mold as the
other tasks. If we didn't have any Coverity usage limitations (think
it's 1 build per day?) and it was required to run on every new commit
in `master`, then I'd say it would be ideal to have that ported into
Cirrus, too, but since that's not the case, I don't have much
preference where it runs except doesn't seem efficient to pay to
bypass the 50 minute Travis time limit anymore just for doing the
Coverity build if GitHub Actions has a free 6 hour time limit and also
supports cron workflows.

I see your point there. A single GitHub action would be enough to do the single-platform build that we need for Coverity.

3. Can you move the travis-job script up into the ci directory instead of leaving it in testing?

Just to keep things a bit more organized while Travis is still being
used in parallel with Cirrus? Sure, but eventually travis-job can get
removed: rewrote/factored bits of that into the other ci/*.sh scripts
for Cirrus to use.

That’s what I was thinking. It seems strange to have it out on an island by itself, even if it’s going to get removed eventually.