New candidate JEP: 369: Migrate to GitHub
Kevin Rushforth
kevin.rushforth at oracle.com
Wed Nov 13 18:34:50 UTC 2019
Hi Mario.
You raise a very valid point in wondering how the Skara tooling and
GitHub workflow will scale from a smaller project like OpenJFX (or
Skara) to a larger project like the JDK or JDK Updates project. There
are two aspects of the scaling that will need to be considered. First
the higher volume may lead to some additional challenges in its own
right. Second, the jdk is made up of many sub-projects each with their
own mailing list, so the tooling will need to manage different reviews
which are targeted for different mailing lists (plural in some cases
where a single PR affects more than one sub-project). I think the Skara
team has a good plan for addressing them.
I would point out that many of the concerns raised have been around the
migration to Git and the GitHub review process itself, where I do think
the experiences of smaller projects like OpenJFK or Skara itself is
relevant.
-- Kevin
On 11/13/2019 10:21 AM, Mario Torre wrote:
> Hi Kevin,
>
> First of all I want to apologise if what I'm about to say may sound
> harsh because it's definitely not my intention to undermine the
> validity of the project in itself, however I fear that this sort of
> experience may apply well for "smaller" projects with one or two
> maintainers doing the most of the merging work.
>
> The OpenJFX project has currently 39 contributors listed on GitHub and
> is quite healthy, but in the last month, according to GitHub (and if I
> read the stats correctly!), you had "only" 7 authors with 13 commits
> to master and an average of 5 to 7 reviews exchanges. This is a lot of
> work already, but is nowhere close to the volume and complexity of
> OpenJDK and of the update projects.
>
> Cheers,
> Mario
>
> Il giorno mer 13 nov 2019 alle ore 17:22 Kevin Rushforth
> <kevin.rushforth at oracle.com> ha scritto:
>> I would like to add my experience with Skara. I lead the OpenJFX project
>> [1] along with Johan Vos. We are one of the early adopters of Skara,
>> having recently moved our official repo to GitHub as an early access
>> test of the Skara tooling and the transition to Git / GitHub. We've been
>> using it exclusively for over a month now, with positive results.
>>
>> Our experience with doing reviews on GitHub goes back even farther.
>> About a year prior to moving our official repo from hg.openjdk.java.net
>> to GitHub, we setup a mirror repo on GitHub that could be used as an
>> alternative to webrev for code reviews. What we found during the past
>> year is that most (but not all) contributors started using the GitHub
>> sandbox as their preferred mode for doing code reviews. We were able to
>> attract several new contributors who might not otherwise have felt they
>> were able to participate. The one downside we had during that year is
>> that we still had to export the commit from Git after the review was
>> finished and import it into a Mercurial changeset. With the official
>> repo being on GitHub, and the Skara tooling enabled, that extra step is
>> gone and we have a smooth workflow.
>>
>> I have found it easier, not harder, to exchange feedback on proposed
>> changes using GitHub -- I think it's much easier to comment on a block
>> of code inline rather than copying and pasting it from the webrev into
>> an email message. One of the nice things about the Skara tooling,
>> though, is that reviewers have their choice. Since review comments are
>> cross-forwarded between the mailing list and the PR, both modes are
>> supported. There are some wrinkles in the tooling, but the Skara team
>> has been very responsive in addressing them.
>>
>> To be sure, there is a bit of a learning curve moving to Git and moving
>> to a pull request model where each developer has a fork of the actual
>> repo, but I think that the benefits outweigh the drawbacks.
>>
>> -- Kevin
>>
>> [1] https://github.com/openjdk/jfx
>>
>>
>> On 11/13/2019 7:00 AM, Erik Helin wrote:
>>> On 11/13/19 3:15 PM, Thomas Stüfe wrote:
>>>> Hi Erik,
>>> Hi Thomas, hope you are doing well!
>>>
>>>> First off thanks a lot to you and Joe for all your hard work.
>>> There are many more than me and Joe that have contributed to Skara :)
>>> In no particular order, a big thanks to:
>>> - Robin Westberg
>>> - Mark Reinhold
>>> - Erik Joelsson
>>> - Tony Squier
>>> - Kevin Rushforth
>>> - Jorn Vernee
>>> - Stanislav Smirnov
>>> - Christian Tornqvist
>>> - Dalibor Topic
>>> - Jeff Dinkins
>>> - Tim Bell
>>> - Marcus Hirt
>>>
>>>> After we last talked about this in February I think this project is
>>>> in good and unbiased hands.
>>>>
>>>> However, I share some of Andrews concerns. He voiced them more
>>>> elaborately than I could, so just some additional thoughts:
>>>>
>>>> When we introduce a new channel of discussion, we risk splitting the
>>>> community and damaging communication between us. We also risk
>>>> off-putting existing developers and lose productivity. The assumption
>>>> with this JEP is therefore that we gain more developers than we lose.
>>> No, the assumption is definitely *not* that we would lose developers.
>>> We will hopefully gain some contributors, but I don't intend to lose any.
>>>
>>>> These points should be clearly reflected under "Risk and
>>>> Assumptions". In my mind these are bigger points even than the risk
>>>> of locking in metadata with a proprietary provider.
>>>>
>>>> One smaller negative effect: mailing lists may receive less developer
>>>> attention. You lose those developers which moves away to GitHub. This
>>>> may lead to people not getting answers as promptly as before on
>>>> non-RFR matters.
>>> A contributor to OpenJDK will still be expected to follow and be
>>> active on the OpenJDK mailing lists, that does not change with Skara
>>> nor with a move to GitHub. For the Skara project iself, this is
>>> something we actively communicate on GitHub, see for example the Skara
>>> README [0] and Skara's contribution guidelines [1] (shown the first
>>> time a GitHub user submits a pull request targeting Skara). If needed
>>> we will communicate this even further by for example using pull
>>> request templates [2] (show every time a pull request is submitted).
>>>
>>> These communication techniques can of course also be utilized for
>>> other projects, e.g. the JDK project, if wanted.
>>>
>>>> The notion to keep both ways (mailing list + GitHub PRs) functional
>>>> and "first class citizens" is great. But I'm afraid that it may turn
>>>> out too brittle to keep up, and that the old side would rot away over
>>>> time. Which would cause a sliding effect - frustrated developers
>>>> switching away from mailing lists to GitHub, which would in turn
>>>> leave less developers on the mailing lists, which makes for even less
>>>> incentive to keep it working.
>>> There is no such thing as "switching away from mailing lists to
>>> GitHub". All activity on GitHub is mirrored on the OpenJDK mailing
>>> lists. And as I stated above, an active OpenJDK contributor is
>>> absolutely expected to follow and be active on the mailing lists.
>>> There is much more activity on the mailing lists besides "RFR" emails:
>>> - design discussion
>>> - call for votes
>>> - announcements
>>> - JEP discussion (such as this one)
>>> - occasional bug reports
>>> - etc.
>>>
>>>> Please do not understand this as me questioning your work - I am sure
>>>> as long as the tools have your attention they will work. But I would
>>>> feel better with a process which is less involved. One can say what
>>>> one wants about mailing lists, but they are low tech and resilient.
>>>>
>>>> So, this is also an assumption - that the tool chain keeps working
>>>> and that the effort spent maintaining it is less than the effort
>>>> spent today on maintaining the old infrastructure (which to be fair
>>>> is done by Oracle alone).
>>> Yes, the Skara tools need to keep on working, that is correct. That
>>> is, as you also state, of course true for all the infrastructure
>>> supporting OpenJDK.
>>>
>>> The Skara tooling has been written with ease of maintenance in mind.
>>> The tools are written in Java, the only dependencies are Gradle for
>>> building and JUnit for testing. We have plenty of unit tests and
>>> integration tests and the bots themselves are written in a polling and
>>> mostly stateless architecture.
>>>
>>> The Skara tooling is also open source [3] and we have already gotten
>>> some very nice community contributions (thanks!).
>>>
>>>> One other risk to mention would be that even if you are happy with
>>>> the way GitHub does represent PR discussions now, GitHub may change
>>>> the interface any time they like. With mailing lists, one is
>>>> independent of these interruptions.
>>> Again, you will not loose the mailing lists :) If you don't like the
>>> GitHub UI, then use one of the several third-party clients and/or
>>> interact via the mailing lists. We will likely discover some kinks
>>> with the bi-directional mailing list synchronization that we will have
>>> to work out, but so far things have been working very well (otherwise
>>> we wouldn't be submitting this JEP).
>>>
>>> Thanks,
>>> Erik
>>>
>>> [0]: https://git.openjdk.java.net/skara#discuss
>>> [1]: https://git.openjdk.java.net/skara/blob/master/CONTRIBUTING.md
>>> [2]:
>>> https://help.github.com/en/github/building-a-strong-community/about-issue-and-pull-request-templates
>>> [3]: https://git.openjdk.java.net/skara
>>>
>>>> -----------
>>>>
>>>> Smaller technical nits:
>>>>
>>>> Changing the subject line of mails depending on the review iterations
>>>> does not play well with most email clients and with mailing list
>>>> archives, which thread mails by subject line. It tears discussions
>>>> apart.
>>>>
>>>> --
>>>>
>>>> "Do not change the OpenJDK Bylaws.
>>>> Do not change the OpenJDK Census."
>>>>
>>>> Should be listed under Non-Goals?
>>>>
>>>> Thank you,
>>>>
>>>> Thomas
>>>>
>>>>
>>>> On Wed, Nov 13, 2019 at 12:42 PM Erik Helin <erik.helin at oracle.com
>>>> <mailto:erik.helin at oracle.com>> wrote:
>>>>
>>>> On 11/13/19 11:36 AM, Andrew Dinn wrote:
>>>> > On 12/11/2019 16:00, mark.reinhold at oracle.com
>>>> <mailto:mark.reinhold at oracle.com> wrote:
>>>> >> https://openjdk.java.net/jeps/369
>>>> > Firstly, thanks to Joe and Erik for producing a very
>>>> comprehensive JEP
>>>> > that addresses so many important questions. There are many
>>>> aspects of
>>>> > this proposed move that are very appealing.
>>>>
>>>> Thanks Andrew for reading through it all and providing feedback!
>>>> Please
>>>> see my replies inline.
>>>>
>>>> > However, I am afraid I am
>>>> > still going to start by expressing my severe reservations
>>>> about one
>>>> > proposed change: replacing email review with GitHub PRs (sorry
>>>> guys :-/).
>>>> >
>>>> > The JEP has this laudable goal:
>>>> >
>>>> > "Ensure that workflows structurally similar to the existing
>>>> e-mail and
>>>> > webrev based workflows are supported."
>>>> >
>>>> > and further qualifies that with:
>>>> >
>>>> > "Today, OpenJDK contributors collaborate via mailing lists,
>>>> they push
>>>> > changes to Mercurial repositories, they test changes via the
>>>> jdk/submit
>>>> > service, and they file bug reports via the JDK Bug System (JBS).
>>>> > Contributors can also make use of several command-line interface
>>>> (CLI)
>>>> > tools, primarily jcheck, webrev, and defpath. This is a
>>>> workflow that
>>>> > many experienced contributors enjoy and find productive. It's
>>>> therefore
>>>> > critical to preserve as much of this workflow as possible if we
>>>> move to
>>>> > an external provider."
>>>> >
>>>> > which one cannot really disagree with.
>>>> >
>>>> > The JEP then explains that:
>>>> >
>>>> > "Reviewers can now discuss the changes in the PR, using multiple
>>>> workflows:
>>>> >
>>>> > By replying to the RFR email sent to the mailing list(s), in
>>>> which
>>>> > case the contents of replies are copied into the PR on GitHub (no
>>>> GitHub
>>>> > account is required);
>>>> > . . .
>>>> > Any comment made in any of the workflows is reflected in all
>>>> of the
>>>> > workflows. One reviewer can add a comment via the mailing list,
>>>> another
>>>> > via the web browser, and a third via the command-line and they
>>>> will all
>>>> > see each others' comments."
>>>> >
>>>> > My experience of GitHub use in the Graal project suggest that
>>>> this is
>>>> > not entirely the full picture. My view derived from that
>>>> experience is
>>>> > that the GitHub PR is very much an inferior medium for review
>>>> > discussion. In particular, it definitely fails to be
>>>> "structurally
>>>> > similar to the existing e-mail and webrev based workflows".
>>>>
>>>> I think the key sentence here is "My experience of GitHub use in the
>>>> Graal project...". Having worked with project on GitHub using Skara
>>>> tooling and projects on GitHub _not_ using Skara tooling, my view is
>>>> that the experiences differs quite a bit.
>>>>
>>>> > Firstly, I'll note that email discussions consist of a tree of
>>>> dependent
>>>> > commentary. That commentary often contains quoted material which,
>>>> > especially when carefully edited (as it normally is), tracks
>>>> relevant
>>>> > context. The structuring that results from this use of the reply
>>>> > hierarchy and quoting permits a given discussion to split into a
>>>> group
>>>> > of related subordinate discussions where required while still
>>>> retaining
>>>> > a continued thread of relevance to a specific review.
>>>> Crucially, most
>>>> > email readers allow such branching discussions to be followed and
>>>> > extended in /parallel/.
>>>> >
>>>> > By contrast discussions in GitHub PRs are essentially linearized
>>>> (modulo
>>>> > a single level of nesting of commentary on top level comments).
>>>> Worse,
>>>> > the presentation forces all commentary to be flattened into a
>>>> single
>>>> > browser pane view. This linear mode of presentation results in a
>>>> severe
>>>> > hindrance to separation of, and attention to, separate concerns.
>>>> It also
>>>> > fails to preserve and display source relationships that
>>>> clarify the
>>>> > provenance and relevance of quoted text i.e. it not only fails to
>>>> record
>>>> > but actually obfuscates important context.
>>>>
>>>> This does not paint the whole picture. You are probably thinking
>>>> of the
>>>> "Conversation" tab in a GitHub pull request which is meant for
>>>> *general*
>>>> discussion of the patch that is not attached to any particular
>>>> change in
>>>> the patch. GitHub's own help documentation [0] states that the
>>>> "Conversation" tab is meant to be used for:
>>>>
>>>> You can comment on a pull request's Conversation tab to leave
>>>> general comments, questions, or props.
>>>>
>>>> You are right that comments in the "Conversation" tab are linearized
>>>> and
>>>> does not support a "tree style" view of comments.
>>>>
>>>> The good news are that the _other_ form of comments available on a
>>>> GitHub pull request, a so-called "review comment" or "file comment"
>>>> [1],
>>>> does support replies in the form of a tree. These comments are filed
>>>> under the "Files changed" tab in a pull request.
>>>>
>>>> Personally I think of the comments in the "Conversation" tab as
>>>> replies
>>>> to the "00" email in a classic patch series email and the "review
>>>> comments" in the "Files changed" tab as replies to the "0X" emails
>>>> actually containing patches. Do you follow?
>>>>
>>>> Now the interesting question is of course how the Skara tooling
>>>> translates these kinds of comments back-and-forth between mailing
>>>> lists
>>>> and GitHub. Here I'm happy to report that the Skara tooling does
>>>> proper
>>>> replying, which will cause the "review comments" created under the
>>>> "Files changed" tab on a pull request to result in a tree-view (in
>>>> email
>>>> clients that support such views).
>>>>
>>>> You can see of some of this work on the openjfx-dev mailing list
>>>> [2].
>>>> Now keep in mind if you are looking at Pipermail rendering of a
>>>> mailing
>>>> list, which lacks some features that most email clients supports
>>>> (such
>>>> as folding quoted text and nested replies beyond three levels). A
>>>> mailing list version of a pull request will in general have the
>>>> following structure:
>>>>
>>>> - RFR: Fix a bug <-- Email describing patch
>>>> - Re: RFR: Fix a bug <-- This is a general comment
>>>> - Re: RFR: Fix a bug <-- This is a reply to the general
>>>> comment
>>>> - Re: [Rev 01]: RFR: Fix a bug <-- The author updated the patch
>>>> - Re: [Rev 01]: RFR: Fix a bug <-- comment from reviewer A
>>>> on 01
>>>> - Re: [Rev 01]: RFR: Fix a bug <-- reply to comment from
>>>> A on 01
>>>> - Re: [Rev 01]: RFR: Fix a bug <-- comment from reviewer B
>>>> on 01
>>>> - Re: [Rev 01]: RFR: Fix a bug <-- reply to comment from
>>>> B on 01
>>>> - Re: [Rev 02]: RFR: Fix a bug <-- The author updated again
>>>> - Re: [Approved] RFR: Fix a bug <-- Reviewer A approved the
>>>> patch
>>>> - Re: [Approved] RFR: Fix a bug <-- Reviewer B approved the
>>>> patch
>>>> - Re: [Integrated] RFR: Fix a bug <-- Author integrated the
>>>> patch
>>>>
>>>> We are tweaking this structure as we get more and more experience
>>>> with
>>>> it, but at the moment I'm fairly satisfied with threading and the
>>>> tree
>>>> layout. If you have any feedback on this structure/layout, then
>>>> we are
>>>> happy to hear it.
>>>>
>>>> Going in the other direction, mailing list -> GitHub, we also try to
>>>> preserve the mailing list structure as much as possible. This is
>>>> actually a harder challenge, since an email is less structured
>>>> compared
>>>> to a comment on GitHub. An example of this direction can be found in
>>>> pull request 231 for the Skara repository [3] where you can see the
>>>> thread (which is a tree) from skara-dev at openjdk.java.net
>>>> <mailto:skara-dev at openjdk.java.net> [4] being
>>>> "flattened" and uses quotation to try to preserve the flow.
>>>>
>>>> > Of course, discussions that remain in email may continue to
>>>> profit from
>>>> > the multi-focus aspect of the current workflow. However, I
>>>> believe that
>>>> > contributions to the discussion via a GitHub PR will inevitably
>>>> bypass
>>>> > and, therefore, invalidate any such structuring contributed
>>>> via the
>>>> > email workflow. Not only do I currently see that effect with
>>>> Graal PRs
>>>> > but I also see the damage it imposes on flow and quality of
>>>> discussion.
>>>> >
>>>> > Secondly, email reviews are conventionally directed to readers
>>>> with
>>>> > different domains of interest and expertise. That sometimes
>>>> requires
>>>> > re-routing a discussion to a different list from the one on which
>>>> it was
>>>> > first initiated. It also sometimes requires posting comments to
>>>> multiple
>>>> > lists, either when discussion is initiated or in mid-stream as
>>>> the scope
>>>> > of discussion develops. Managing distribution when using emails
>>>> lists is
>>>> > both easily achieved and well understood (n.b. that's not to
>>>> imply that
>>>> > deciding which list to send to is easy).
>>>>
>>>> Here we are currently working on the /cc command that can be used
>>>> in a
>>>> comment on pull request, for example:
>>>>
>>>> /cc build-dev hotspot-dev
>>>>
>>>> > The JEP suggests that discussions raised via PRs will be routed
>>>> to lists
>>>> > 1) derived from the affected files and/or 2) specified by whoever
>>>> raises
>>>> > the PR. It doesn't explain how (whether) target lists are to be
>>>> (may be)
>>>> > updated as review progresses. It also doesn't state clearly
>>>> whether
>>>> > whoever raises the PR or comments on it will have the ability to
>>>> > override those default choices (I fully expect that option
>>>> will be
>>>> > necessary in some cases).
>>>>
>>>> Agree, see the command above that we are working on :)
>>>>
>>>> > I need to stress that these concerns should not be considered as
>>>> minor
>>>> > details of how the project operates. Easy management of the
>>>> flow and
>>>> > direction of discussion is critical to being able to consume and
>>>> respond
>>>> > to commentary at the rate needed to keep up with a project as
>>>> large and
>>>> > rapidly changing as OpenJDK currently is. Given the central
>>>> importance
>>>> > of our review process to ensuring the health of the project
>>>> and its
>>>> > products I think it is right to be very concerned about the
>>>> potential
>>>> > for problems caused by this switch of medium.
>>>>
>>>> I hope that is does not come across that we are taking mailing list
>>>> interopability as a minor detail? I think it is fair to say that
>>>> this is
>>>> the Skara feature we have spent the most time working on and are
>>>> constantly improving in order to provide a good experience.
>>>>
>>>> Thanks,
>>>> Erik
>>>>
>>>> [0]:
>>>> https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/commenting-on-a-pull-request
>>>> [1]:
>>>> https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/reviewing-proposed-changes-in-a-pull-request
>>>> [2]:
>>>> https://mail.openjdk.java.net/pipermail/openjfx-dev/2019-November/thread.html
>>>> [3]: https://git.openjdk.java.net/skara/pull/231
>>>> [4]:
>>>> https://mail.openjdk.java.net/pipermail/skara-dev/2019-October/000998.html
>>>>
>>>> > regards,
>>>> >
>>>> >
>>>> > Andrew Dinn
>>>> > -----------
>>>> > Senior Principal Software Engineer
>>>> > Red Hat UK Ltd
>>>> > Registered in England and Wales under Company Registration No.
>>>> 03798903
>>>> > Directors: Michael Cunningham, Michael ("Mike") O'Neill
>>>> >
>>>>
>
More information about the discuss
mailing list