Proposal: use backports for working with release stabilization repositories

Wed Apr 12 10:23:36 UTC 2023

> On 12 Apr 2023, at 11:26, Lindenmaier, Goetz <goetz.lindenmaier at sap.com> wrote:
> 
> Hi,

Hi Goetz,

thanks for providing feedback!

> I see two problems with this approach:
> 
> Usually, fixes in the stabilization repo are time critical.
> One wants to fix the issue before the next tag to catch 
> the next round of testing or before the
> next step in rampdown. Doing the extra cycle via the 
> main repos delays this.

I have to disagree here, in my opinion it is more important
that fixes that go into the release stabilization repository
are well tested - you do not want to introduce additional
issues and churn in the release stabilization repository due
to a bugfix that wasn't properly tested before it was
integrated.

In my opinion it is an advantage that with this proposed model
you can integrate a bugfix into the main-line repository, let
it "soak" for a while to allow the bugfix to get proper testing
and then backport the bugfix to the release stabilization
repository. This at least increases confidence that the bugfix
itself is correct (the backport of the bugfix will of course
require proper testing as well before being integrated into the
release stabilization repository).

> Second, what Jesper is doing is a quite simple, repetitive
> task.  It is quite unlikely that he breaks something.

We had multiple bugs introduced due to incorrect merges in
the main-line repository during the rampdown of JDK 20. We
also have numerous issues in JBS due to contributors
misunderstanding where to push and what should be done in
JBS with the current process.

So no, it is not unlikely.

> If so, 
> it is the main repo and not the stabilization repo, which is
> less critical.  The new process though loads a burden on all 
> developers, and errors or forgotten backports will affect the repo
> that is currently under stabilization.

I analyzed all the commits that has gone into to both the main-line
repository and release stabilization repositories since JDK 16. Of
those contributors that do integrate work into a release stabilization
repostory, they do on average 2-3 commits per contributor per release.
99% of contributors that do integrate work into a release stabilization
repository do less than 10 commits to the release stabilization repository
for a given release.

If the backport is clean (which it very likely will be) then Skara's "/backport"
command automates almost all of the work. The overhead of doing 2 - 3 clean
backports per release with the help of Skara's tooling is in my opinion low
(we are talking about minutes per backport). If the backport is not clean, well,
then the developer that did the original work is the best person to resolve
those merge conflicts.

With regards to missing to backport a bugfix, I'm confident that the JBS query
described in the proposal coupled with the reminders sent to jdk-dev will be enough.
If it is not and this becomes an issue then we can introduce a heavier process,
but I don't want to start out with a heavy process for something that most likely
can be solved with a fairly lightweight process.

> Therefore I think the old process is better for the 
> quality of the stabilization repository.

If I shared that thought I wouldn't have sent out this proposal :)

I've thought about this process for quite some time and have done fairly extensive
analysis of the work going into both the main-line repository and the release
stabilization repository during the rampdown phases for a release. My conclusion is
this new proposed process which I believe will improve the quality of both the release
stabilization repository and the main-line repository.

Thanks,
Erik

> Best regards,
>  Goetz.
> 
> 
> 
> 
> 
> 
> 
> 
> 
>> -----Original Message-----
>> From: jdk-dev <jdk-dev-retn at openjdk.org> On Behalf Of Erik Duveblad
>> Sent: Tuesday, April 11, 2023 2:55 PM
>> To: jdk-dev at openjdk.org
>> Subject: Proposal: use backports for working with release stabilization
>> repositories
>> 
>> Hi all,
>> 
>> this is a proposal to change the process for how we work with commits that
>> should be applied to _both_ the main-line repository (jdk [0]) and a release
>> stabilization repository (e.g. jdk19 [1], jdk20 [2]).
>> 
>> The current process we are using today consist of contributors first creating
>> a pull request targeting the release stabilization repository (e.g. jdk20).
>> Jesper will then periodically merge changes from the release stabilization
>> repository to the main-line (jdk) repository. If the merge contains
>> commits that should *only* be applied to the release stabilization repository,
>> then those commits have to be backed out [3] from the main-line repository.
>> The process of Jesper periodically merging changes from the release
>> stabilization repository to the main-line repository is sometimes referred to as
>> "forward porting". For more information on the current process, see the
>> section on "Forward Ports" in the OpenJDK Developer's Guide [4].
>> 
>> I propose that we change to use backports for working with the release
>> stabilization repositories. With this new process contributors would first
>> create a pull request targeting the main-line (jdk) repository. Once the
>> pull request has been reviewed and integrated into the main-line repository
>> contributors will backport the commit to the release stabilization repository
>> (e.g. jdk20). The backport can be created in numerous ways but most
>> contributors will likely use Skara's [5] "/backport" command [6] to create the
>> backport. For changes that should *only* go into the release stabilization
>> repository, contributors will create a pull request targeting the release
>> stabilization repository. Many of you will recognize this process as being
>> essentially the same one that is being used for backporting changes to update
>> releases (modulo JBS labels, approvals, etc.).
>> 
>> All the relevant Skara tooling supporting this proposal has been in place for a
>> long time and is being used by other OpenJDK projects. With this new process
>> we no
>> longer have to care about backing out release stabilization specific commits
>> from the main-line repository, there is a much lower risk of issues being
>> introduced
>> due to incorrect merges and the backport process has proven itself when used
>> by
>> other OpenJDK projects.
>> 
>> I suggest that we adopt this proposal for JDK 21.
>> 
>> Please see below for some common questions and answers:
>> - Q: How can we ensure that no contributor forgets to backport a commit to
>> the
>>     release stabilization repository?
>> - A: An e-mail reminder will be sent to jdk-dev [7] prior to entering the
>>     rampdown phase two (RDP 2) [8] and release candidate (RC) [8] phases. The
>>     e-mail will contain a link to a JBS [6] query that contributors can click to
>>     view issues that they have fixed in the main-line repository that are
>>     applicable to being backported.
>> 
>>     For example, prior to entering RDP 2 for a release an e-mail will be sent
>>     to jdk-dev with a JBS query that shows a user the P3 issues he or she
>>     has fixed in the main-line repository since the release stabilization
>>     repository was created that has *not* been backported to the release
>>     stabilization repository. Every contributor can read through their own
>>     list to verify that they haven't forgotten to backport any P3 issues
>>     before the release enters RDP 2.
>> 
>> - Q: What happens with tags created in the release stabilization repository?
>> - A: They will be pushed to the main-line repository by the Skara bots. Note
>> that
>>     only the tags, and the commits they refer to, will be pushed to the main-line
>>     repository. The commits referred to by the tags will *not* be merged into
>> the
>>     "master" branch in the main-line repository.
>> 
>> - Q: Will "clean" backports [10] have to be re-reviewed?
>> - A: Yes. Skara [5] has a feature where "clean" backports can be configured
>>     (per repository) to be allowed to be integrated without a review. This is
>>     feature is utilized by some projects in OpenJDK. For the JDK project and for
>>     release stabilization repositories this feature will be turned off, meaning
>>     that all backports will require reviews (regardless whether they are
>>     "clean" or not).
>> 
>> - Q: Can someone other than the original contributor do the backport?
>> - A: Yes, this is fully supported by Skara. This often happens when an developer
>>     sponsors a bugfix from a newer contributor who is not yet a committer, and
>>     that bugfix should be backported to an older release. In that case the
>>     sponsor is often the one doing the backport.
>> 
>> If this proposal is accepted then information about this process will be added
>> to the OpenJDK Developer's Guide [11] and to JEP 3 [8]. Please let me know if
>> you have
>> any feedback, comments or questions!
>> 
>> Thanks,
>> Erik
>> 
>> [0]:  https://git.openjdk.org/jdk
>> [1]:  https://git.openjdk.org/jdk19
>> [2]:  https://git.openjdk.org/jdk20
>> [3]:  https://openjdk.org/guide/#backing-out-a-change
>> [4]:  https://openjdk.org/guide/#forward-ports
>> [5]:  https://openjdk.org/projects/skara/
>> [6]:
>> https://wiki.openjdk.org/display/SKARA/Pull+Request+Commands#PullReques
>> tCommands-/backport
>> [7]:  jdk-dev at openjdk.org
>> [8]:  https://openjdk.org/jeps/3
>> [9]:  https://bugs.openjdk.org
>> [10]: https://wiki.openjdk.org/display/SKARA/Backports#Backports-
>> Terminology
>> [11]: https://openjdk.org/guide
> 