Moving to GitHub?

Development
1

This has been coming up in various contexts & subgroups of people, and
I wanted to send it out as a proposal to gather some broader feedback:
Do we want to move Bro's git repositories and tickets to GitHub?

Currently we're hosting our Git repositories on our own server at
git.bro.org; from there, we mirror them to GitHub. For issue tracking,
we use JIRA at tracker.bro.org. Much of all this is a legacy setup in
some sense. I believe it would simplify life for both users &
developers if we moved to GitHub as the authoratative place for both
code and tickets.

More specifically, I propose that we:

     1. Turn the current git repository mirrors on Bro (Zeek) Network Monitoring Project · GitHub into
        authoritative source for all Bro code. Then we discontinue
        git.bro.org. We can set up up some notification system there
        instead that points people still using the old URLs to GitHub.

     2. Switch to using GitHub as our primary issue tracker. Giving
        the large amount of old tickets in JIRA, I think we should
        just start from scratch there, porting over just the most
        recent pending tickets. We'd keep the JIRA running in
        read-only mode so that we don't loose the history and can
        always refer back to old tickets.

     3. Switch to a mostly PR-based development model (i.e., no more
        merge requests tracked separately through tickets), and also
        use GitHub for code review & feedback.

     4. Make sure it all integrates nicely with Travis CI (which has
        already been set up recently).

About the only downside I see is that emailing out our standard commit
notifications won't be quite as smooth anymore: we'd still get them,
but with a bit of delay as some system would need to be polling for
changes, rather than being triggered immediately. Not much of a
problem I think (and with some additional work, we could make them
push-triggered, too; but probably not worth it).

What do people think? Any support, or concerns?

Robin

2

What do people think? Any support, or concerns?

I am in favor. The only thing I would miss are the immediate change notifications by email - I really like those...

Johanna

3

I am in favor. I would like to still maintain the the Jira and wiki for a couple months until we finish up some work. Really just the BPM tickets.

4

Big thumbs up.

5

I too would miss the commit / change notifications, however, I think that this can be set up in GitHub in some way. We use Slack / Email internally for commit notifications. For Slack, we use webhooks to send notifications.

Email commit notifications are actually a builtin integration in GitHub... You can set them up under Settings -> Integrations & Services -> Add service -> Email.

    Message received from outside the Battelle network. Carefully examine it before you open any links or attachments.
    
    > What do people think? Any support, or concerns?
    
    I am in favor. The only thing I would miss are the immediate change
    notifications by email - I really like those...
    
    Johanna

6

Well... in looking at how I set this up internally, I now realize that this feature is being EOL and will no longer work as of January 31, 2019. I guess it would have to be webhooks.

    I too would miss the commit / change notifications, however, I think that this can be set up in GitHub in some way. We use Slack / Email internally for commit notifications. For Slack, we use webhooks to send notifications.
    
    Email commit notifications are actually a builtin integration in GitHub... You can set them up under Settings -> Integrations & Services -> Add service -> Email.
    
        Message received from outside the Battelle network. Carefully examine it before you open any links or attachments.
        
        > What do people think? Any support, or concerns?
        
        I am in favor. The only thing I would miss are the immediate change
        notifications by email - I really like those...
        
        Johanna

7

We can still get the same email notifications as today (which have a
bit more information that GitHub's standard ones), they will just come
with a little bit of a delay (within 10-15 minutes should be
reasonable). And if we want we can trigger that through webbhooks,
too, for immediate notification, would just need a bit of work to get
it set up.

Robin

8

Yeah, generally in favor with some comments:

* For porting over JIRA tickets to GitHub, "most recent" doesn't seem like a good metric to use. e.g. BIT-1829 (pcap triggering assertion in binpac) seems kind of important and not something I'd want to have lost in the move, although it had no activity in almost a year. So, I think it's worth it to be more comprehensive here, and as long as someone is going through to review all the tickets, they may as well just port all the older ones that are still valid over to GitHub. (Yeah, I guess I'm volunteering).

* One thing I did like about using JIRA for merge requests is that I could make a single ticket and just say I have a given branch name in a bunch of repos that are ready for review/merge. I find myself in that situation quite often, actually, so transitioning to GitHub PRs, I wonder if we'd want a PR to be created against each individual repo? Seems a bit much in terms of overhead. Alternatively, could still create a GH issue and just say "please review branch foo in repos X, Y, Z and merge them". Or else create a single PR in the "root" repo and mention in the PR that the same branch in child submodules also exists and needs merging. That may play better with Travis CI integration even, although maybe not in the case where you need to change things in the "external testing" repos, which are not connected to bro via a submodule.

- Jon

9

I like the idea. It'll be nice to have one process for handling everything instead of the multiple avenues for tickets and merge requests that we have today.

It seems that we should even be able to solve the issue with the immediate diff emails by using webhooks with an AWS lambda function. I'm not sure if we'd be able to do all of the git work from that though.

   .Seth

10

* For porting over JIRA tickets to GitHub, "most recent" doesn't seem like a
good metric to use.

Agree. :slight_smile:

they may as well just port all the older ones that are still valid
over to GitHub.

That may be a bit too broad though. How about "still valid and either
(1) quite important or (2) something we expect will be addresses
reasonably soon"? We have many old tickets that are technically still
valid but unlikely to see any work anytime soon (otherwise they would
have been addressed already), and I'm worried that they would just add
noise without value. The old tickets won't go away, the JIRA will
remain. If something becomes relevant/active, we can always bring it
over at that time.

I find myself in that situation quite often, actually, so
transitioning to GitHub PRs, I wonder if we'd want a PR to be created
against each individual repo?

Good point. Creating just one root PR that mentions the others sounds
good to me for such cases.

Robin

11

That may be a bit too broad though. How about "still valid and either
(1) quite important or (2) something we expect will be addresses
reasonably soon"? We have many old tickets that are technically still
valid but unlikely to see any work anytime soon (otherwise they would
have been addressed already), and I'm worried that they would just add
noise without value.

That sounds like a general concern about project/ticket organization and management. What if valid-but-old tickets were moved into GitHub with a simple "backlog" tag that you can filter out? Or we could take the opportunity to be better about sorting on other categories as well (bug/feature/etc).

There's also the possibility to start a project board page [1] to aid in visual organization of tasks/issues and further reduce perceived noise. See [2] for an example of what one of those pages could look like.

The old tickets won't go away, the JIRA will
remain. If something becomes relevant/active, we can always bring it
over at that time.

Keeping the JIRA around as a backlog like that decreases focus/visibility -- if we want to be optimistic about community contributions and bug fixes, we'd have to keep calling attention to search in two locations, GH and JIRA, instead of just one. We'd also need to stay on top of maintaining JIRA to be relatively in sync with anything that gets resolved in GH else keeping JIRA around may become more liability than useful as it gets harder and harder to tell if something there has actually been addressed.

Doing a half-hearted effort to migrate tickets from JIRA undermines the goal of having an authoritative/central location for all code + tickets. Can we instead try to deal with it once and for all?

- Jon

[1] Projects · bro · GitHub
[2] Backlog · GitHub

12

What I was envisioning is more or less a clean slate: we'd migrate
over a few tickets, but essentially we'd start with an empty list. I
realize that sounds pretty harsh. However, I hardly ever see any
activity on older tickets in JIRA, and I generally believe that the
less open tickets a tracker has, the easier it is for people to
understand what's actually relevant and worth spending cycles on.
Tagging tickets may help, but in the end if everybody just filters 95%
out all the time anyways, I'm not sure what the value is.

That said, I'm open to a real porting effort if people do believe it's
helpful to get all the JIRA tickets into GitHub. What do others think?

Robin

13

Start clean for BIT. Unless it is marked critical, I don’t think it needs to go over. If people have tickets of their own they want to move over, it should not be too hard to manually recreate a couple.

And after BroCon I think we kill the Jira and wiki completely. We should be done with the project management and event tickets by then.

14

I am also more in favor of starting clean and manually letting people move tickets that they think are important over.

But - currently there is a lot in the tracker that are nice to have or potential problems that I do not ever see getting addressed.

Johanna

15

That said, I’m open to a real porting effort if people do believe it’s
helpful to get all the JIRA tickets into GitHub. What do others think?

Having the historical tickets available are useful for searching to see if someone else had a similar issue and if there’s a possible work-around.

How about a solution somewhere in the middle - push all the tickets over but mark the older/non-critical as ‘won’t fix’. They’ll come up in searches and can more easily be brought back alive if needed.

…alan

16

What I was envisioning is more or less a clean slate: we'd migrate
over a few tickets, but essentially we'd start with an empty list. I
realize that sounds pretty harsh. However, I hardly ever see any
activity on older tickets in JIRA, and I generally believe that the
less open tickets a tracker has, the easier it is for people to
understand what's actually relevant and worth spending cycles on.

I see those as independent issues:

(1) There's many open tickets: you solve that by actually addressing the tickets

(2) People don't know what tickets are relevant: you solve that by organizing (and maintaining) them according to relevance

My take is that there's been a lack of effort in (2) and if that's not solved, starting with a clean slate on GitHub now only means it's likely to eventually end up in the same situation as JIRA later. What then? Move to another tracker again?

Tagging tickets may help, but in the end if everybody just filters 95%
out all the time anyways, I'm not sure what the value is.

The value is actually having a central place for all tickets and knowing there won't be ongoing hassle with keeping JIRA updated.

Or in reverse, I'm not sure what the value of keeping JIRA open at all is -- we either have to acknowledge its potential to go out of sync with GH in terms of duplicate reports, which makes it more of a hindrance IMO, or we still have to put effort to maintain it in addition to GH.

That said, I'm open to a real porting effort if people do believe it's
helpful to get all the JIRA tickets into GitHub. What do others think?

Pedantically, I'm not saying "port all JIRA tickets", I still want just the "valid" ones whatever we decide that to mean. So more clarifications on potential porting criteria:

* Someone is likely to report the same problem again
* There's clear directions on how to reproduce an undesired behavior
* There been a proposed plan of action recently

And many tickets can be ruled out:

* Vague feature requests
* Not enough details / difficult to reproduce
* Exceptionally old proposals / plans

My plea is: extend the porting criteria beyond "important" and "recent", perform a review of all existing tickets, and, if JIRA is going to stick around as a read-only archive, leave it in a coherent final state.

- Jon

17

* Someone is likely to report the same problem again
* There's clear directions on how to reproduce an undesired behavior
* There been a proposed plan of action recently

And many tickets can be ruled out:

* Vague feature requests
* Not enough details / difficult to reproduce
* Exceptionally old proposals / plans

Yeah, I'm on board with these. I'd probably interpret them more
conservatively than you towards closing more tickets, but that's fine.
As you have volunteered to take this one, I'd say you get to make the
call: just go through and port over what you think makes sense along
those lines. As one additional piece, let's also think about some tags
to use for classifying tickets, including one for what's good tasks
for newcomers who want to get into the code.

(In principle I also like Alan's suggestion of moving everything over
and then just close them out so that the history remains. But I'm
afraid that couldn't be automated easily and would then just be too
much work.)

starting with a clean slate on GitHub now only means it's likely to
eventually end up in the same situation as JIRA later. What then?
Move to another tracker again?

Doesn't need to be as drastic: as some people here can confirm, I
have no problem doing extensive sweeps if things get too overwhelming. :slight_smile:

But yes, point taken, my hope is that we can stay on top of things on
the new tracker and make an effort to get stuff addressed and
resolved. We'll see. :slight_smile:

Robin