Proposal: use backports for working with release stabilization repositories

Wed Apr 12 22:38:41 UTC 2023

I agree with Erik's points. We internally use a very similar model for 
other releases and have found that in almost all cases, it's much better 
for the stability of the release undergoing stabilization to push to the 
mainline first and then do the backport.

Most fixes can benefit from a little soak time in the mainline before 
backporting them. It doesn't have to be a "one size fits all" (so no one 
is suggesting a fixed amount of time) -- simple fixes don't need much 
bake time, but even having a short time between the integration of the 
mainline fix and the backport helps avoid the occasional CI build or 
failure in the stabilization release, where you can least afford a failure.

In the rare case where a stopper bug really does need to get into the 
stabilization repo ASAP, a developer can send both the mainline and the 
stabilization PR at the same time and have them reviewed in parallel.

I would only add one additional recommendation to what Erik has said: 
Skara now has the ability to require reviews of even clean backports for 
some repos. I think we should adopt that for the feature release repos 
to avoid fixes going into the stabilization repo without having a 
Reviewer approve it. I think this is a good idea even in the absence of 
this proposed change (since nothing prevents a developer today from 
taking a fix from mainline and backporting it without further review), 
but will become even more important if and when this is adopted, and 
virtually all backports are done as Skara backports.

-- Kevin

On 4/12/2023 3:23 AM, Erik Duveblad wrote:
>> On 12 Apr 2023, at 11:26, Lindenmaier, Goetz <goetz.lindenmaier at sap.com> wrote:
>>
>> Hi,
> Hi Goetz,
>
> thanks for providing feedback!
>
>> I see two problems with this approach:
>>
>> Usually, fixes in the stabilization repo are time critical.
>> One wants to fix the issue before the next tag to catch
>> the next round of testing or before the
>> next step in rampdown. Doing the extra cycle via the
>> main repos delays this.
> I have to disagree here, in my opinion it is more important
> that fixes that go into the release stabilization repository
> are well tested - you do not want to introduce additional
> issues and churn in the release stabilization repository due
> to a bugfix that wasn't properly tested before it was
> integrated.
>
> In my opinion it is an advantage that with this proposed model
> you can integrate a bugfix into the main-line repository, let
> it "soak" for a while to allow the bugfix to get proper testing
> and then backport the bugfix to the release stabilization
> repository. This at least increases confidence that the bugfix
> itself is correct (the backport of the bugfix will of course
> require proper testing as well before being integrated into the
> release stabilization repository).
>
>
>> Second, what Jesper is doing is a quite simple, repetitive
>> task.  It is quite unlikely that he breaks something.
> We had multiple bugs introduced due to incorrect merges in
> the main-line repository during the rampdown of JDK 20. We
> also have numerous issues in JBS due to contributors
> misunderstanding where to push and what should be done in
> JBS with the current process.
>
> So no, it is not unlikely.
>
>> If so,
>> it is the main repo and not the stabilization repo, which is
>> less critical.  The new process though loads a burden on all
>> developers, and errors or forgotten backports will affect the repo
>> that is currently under stabilization.
> I analyzed all the commits that has gone into to both the main-line
> repository and release stabilization repositories since JDK 16. Of
> those contributors that do integrate work into a release stabilization
> repostory, they do on average 2-3 commits per contributor per release.
> 99% of contributors that do integrate work into a release stabilization
> repository do less than 10 commits to the release stabilization repository
> for a given release.
>
> If the backport is clean (which it very likely will be) then Skara's "/backport"
> command automates almost all of the work. The overhead of doing 2 - 3 clean
> backports per release with the help of Skara's tooling is in my opinion low
> (we are talking about minutes per backport). If the backport is not clean, well,
> then the developer that did the original work is the best person to resolve
> those merge conflicts.
>
> With regards to missing to backport a bugfix, I'm confident that the JBS query
> described in the proposal coupled with the reminders sent to jdk-dev will be enough.
> If it is not and this becomes an issue then we can introduce a heavier process,
> but I don't want to start out with a heavy process for something that most likely
> can be solved with a fairly lightweight process.
>
>> Therefore I think the old process is better for the
>> quality of the stabilization repository.
> If I shared that thought I wouldn't have sent out this proposal :)
>
> I've thought about this process for quite some time and have done fairly extensive
> analysis of the work going into both the main-line repository and the release
> stabilization repository during the rampdown phases for a release. My conclusion is
> this new proposed process which I believe will improve the quality of both the release
> stabilization repository and the main-line repository.
>
> Thanks,
> Erik
>
>> Best regards,
>>   Goetz.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> -----Original Message-----
>>> From: jdk-dev <jdk-dev-retn at openjdk.org> On Behalf Of Erik Duveblad
>>> Sent: Tuesday, April 11, 2023 2:55 PM
>>> To: jdk-dev at openjdk.org
>>> Subject: Proposal: use backports for working with release stabilization
>>> repositories
>>>
>>> Hi all,
>>>
>>> this is a proposal to change the process for how we work with commits that
>>> should be applied to _both_ the main-line repository (jdk [0]) and a release
>>> stabilization repository (e.g. jdk19 [1], jdk20 [2]).
>>>
>>> The current process we are using today consist of contributors first creating
>>> a pull request targeting the release stabilization repository (e.g. jdk20).
>>> Jesper will then periodically merge changes from the release stabilization
>>> repository to the main-line (jdk) repository. If the merge contains
>>> commits that should *only* be applied to the release stabilization repository,
>>> then those commits have to be backed out [3] from the main-line repository.
>>> The process of Jesper periodically merging changes from the release
>>> stabilization repository to the main-line repository is sometimes referred to as
>>> "forward porting". For more information on the current process, see the
>>> section on "Forward Ports" in the OpenJDK Developer's Guide [4].
>>>
>>> I propose that we change to use backports for working with the release
>>> stabilization repositories. With this new process contributors would first
>>> create a pull request targeting the main-line (jdk) repository. Once the
>>> pull request has been reviewed and integrated into the main-line repository
>>> contributors will backport the commit to the release stabilization repository
>>> (e.g. jdk20). The backport can be created in numerous ways but most
>>> contributors will likely use Skara's [5] "/backport" command [6] to create the
>>> backport. For changes that should *only* go into the release stabilization
>>> repository, contributors will create a pull request targeting the release
>>> stabilization repository. Many of you will recognize this process as being
>>> essentially the same one that is being used for backporting changes to update
>>> releases (modulo JBS labels, approvals, etc.).
>>>
>>> All the relevant Skara tooling supporting this proposal has been in place for a
>>> long time and is being used by other OpenJDK projects. With this new process
>>> we no
>>> longer have to care about backing out release stabilization specific commits
>>> from the main-line repository, there is a much lower risk of issues being
>>> introduced
>>> due to incorrect merges and the backport process has proven itself when used
>>> by
>>> other OpenJDK projects.
>>>
>>> I suggest that we adopt this proposal for JDK 21.
>>>
>>> Please see below for some common questions and answers:
>>> - Q: How can we ensure that no contributor forgets to backport a commit to
>>> the
>>>      release stabilization repository?
>>> - A: An e-mail reminder will be sent to jdk-dev [7] prior to entering the
>>>      rampdown phase two (RDP 2) [8] and release candidate (RC) [8] phases. The
>>>      e-mail will contain a link to a JBS [6] query that contributors can click to
>>>      view issues that they have fixed in the main-line repository that are
>>>      applicable to being backported.
>>>
>>>      For example, prior to entering RDP 2 for a release an e-mail will be sent
>>>      to jdk-dev with a JBS query that shows a user the P3 issues he or she
>>>      has fixed in the main-line repository since the release stabilization
>>>      repository was created that has *not* been backported to the release
>>>      stabilization repository. Every contributor can read through their own
>>>      list to verify that they haven't forgotten to backport any P3 issues
>>>      before the release enters RDP 2.
>>>
>>> - Q: What happens with tags created in the release stabilization repository?
>>> - A: They will be pushed to the main-line repository by the Skara bots. Note
>>> that
>>>      only the tags, and the commits they refer to, will be pushed to the main-line
>>>      repository. The commits referred to by the tags will *not* be merged into
>>> the
>>>      "master" branch in the main-line repository.
>>>
>>> - Q: Will "clean" backports [10] have to be re-reviewed?
>>> - A: Yes. Skara [5] has a feature where "clean" backports can be configured
>>>      (per repository) to be allowed to be integrated without a review. This is
>>>      feature is utilized by some projects in OpenJDK. For the JDK project and for
>>>      release stabilization repositories this feature will be turned off, meaning
>>>      that all backports will require reviews (regardless whether they are
>>>      "clean" or not).
>>>
>>> - Q: Can someone other than the original contributor do the backport?
>>> - A: Yes, this is fully supported by Skara. This often happens when an developer
>>>      sponsors a bugfix from a newer contributor who is not yet a committer, and
>>>      that bugfix should be backported to an older release. In that case the
>>>      sponsor is often the one doing the backport.
>>>
>>> If this proposal is accepted then information about this process will be added
>>> to the OpenJDK Developer's Guide [11] and to JEP 3 [8]. Please let me know if
>>> you have
>>> any feedback, comments or questions!
>>>
>>> Thanks,
>>> Erik
>>>
>>> [0]:  https://git.openjdk.org/jdk
>>> [1]:  https://git.openjdk.org/jdk19
>>> [2]:  https://git.openjdk.org/jdk20
>>> [3]:  https://openjdk.org/guide/#backing-out-a-change
>>> [4]:  https://openjdk.org/guide/#forward-ports
>>> [5]:  https://openjdk.org/projects/skara/
>>> [6]:
>>> https://wiki.openjdk.org/display/SKARA/Pull+Request+Commands#PullReques
>>> tCommands-/backport
>>> [7]:  jdk-dev at openjdk.org
>>> [8]:  https://openjdk.org/jeps/3
>>> [9]:  https://bugs.openjdk.org
>>> [10]: https://wiki.openjdk.org/display/SKARA/Backports#Backports-
>>> Terminology
>>> [11]: https://openjdk.org/guide