[11u] Proposal: Switch jdk11u development to Git/Skara with 11.0.13 cycle

[Vladimir Kempik]

Hello
so the old process, with hg:
if backport applies cleanly - do request to backport via jbs, once approved - push it to the repo, preserving original patch metadata (author, etc)

new process with skara:
if backport appliess cleanly - create pull request, if bot decides its clean - can be integrated without review, then get approval via jbs, then integrate
if backport does not apply cleanly - create pull request, wait for it to be reviewed, then get approval via jbs, then integrate

issue1: it’s bot who decides if this a clean backport or not, almost clean == not clean
issue2 ( small one): clean backport is pushed under the name of backporter, not original author
when we only started with skara on jdk13u-dev, clean backports mechanics wasn’t existing yet, so every backport had to be reviewed, slowing us, now it’s a bit easier.

Skara is generally good for backports now. You may want to try to backport something into jdk13u-dev to test it yourself before you switch jdk11u-dev to git.

Regards, Vladimir.

> 17 февр. 2021 г., в 19:18, Vladimir Kempik <vkempik at azul.com> написал(а):
> 
> Hello
> 
> We have observed one issue on jdk13u-dev after switching to skara: doing clean backport becomes a lot more complicated than with hg, as result it slowed our backporting pace.
> 
> Regards, Vladimir.
> 
> Volker Simonis <volker.simonis at gmail.com> 17 февраля 2021 г. 16:54:46 написал:
> 
> Hi Erik^2 :)
> 
> thank you so much for the detailed description. I'm especially excited
> that you've already published fully consolidated mono-repos for jdk 6
> to 9. That's really very much appreciated!
> 
> Now that the heavy lifting has been done, I hope we can seriously
> consider migrating 8u to Git as well. @OpenJDK8u-Maintainers - what
> blockers are you still seeing? Is there anything the community can
> help with to speedup the migration?
> 
> Thank you and best regards,
> Volker
> 
> On Tue, Feb 16, 2021 at 8:31 PM <erik.joelsson at oracle.com> wrote:
> 
> Hello,
> 
> As you may have seen, Erik Duveblad just posted about our prototype
> conversions on skara-dev [1]. These are based on the same kind of
> forest->mono repo conversion, followed by hg->git conversion that
> mainline went through.
> 
> I have been working for a while now on adapting the scripts from JEP 296
> to work for the old update releases. This has required more work than I
> first anticipated as the problem now requires a more generalized
> solution, but I believe I now have something that works reasonably well.
> If you are curious on the details, please continue reading.
> 
> Volker's description of the processes is pretty accurate on a high
> level. The main difference in 8u compared to a main release is that the
> tags are not sequential in history. This means that we can't transplant
> (or splice as it's called in Mercurial terms) changes as freely on top
> of each tagged change. In the original JDK 10 consolidation, we had a
> handful of changes that weren't linear (around the transition between
> JDK 9 and 10), and I handled them differently (basically just merging
> them together to recreate the point in history, but not splicing
> anything on top afterwards). Since they were only a handful, I could
> just list them out explicitly in the script. For 8u, a majority of the
> tags are non linear, so a big problem was creating an algorithm that
> would automatically figure out a reasonable line of tags in sequence
> that I could use to splice changes on top of.
> 
> Another big problem is that there are often tags merged together in
> different order in different repos. When this happens, the script needs
> to figure out the best order to do the conversion. I have tried to
> automate this process as much as possible. The ordering is sensitive
> when some of the tags are used to splice changes on top of them. There
> are also cases of tags missing completely.
> 
> In the end, the state of the mono repo at each tag is verified to be
> exactly the same as the forest of repos at each tag. The state of
> changes between tags is however less predictable in 8u compared to
> mainline due to the large amount of non linear tags.
> 
> For the extra curious, here is what the current algorithm for choosing
> the "splice tags" looks like:
> 
> 1. Start with the first tag
> 
> 2. Continue with each consecutive build for current release until a gap
> in build numbers is detected (we typically have update release builds
> b30 that probably aren't that interesting)
> 
> 3. Find the next release in order and then find the first tag in that
> release that is also a descendant of the previous splice tag in every repo.
> 
> 4. Repeat from 2 until all tags have been considered.
> 
> /Erik
> 
> [1]
> https://mail.openjdk.java.net/pipermail/skara-dev/2021-February/004228.html
> 
> On 2021-02-12 13:33, Volker Simonis wrote:
> On Fri, Feb 12, 2021 at 6:40 PM Andrew Dinn <adinn at redhat.com> wrote:
> On 12/02/2021 16:37, Severin Gehwolf wrote:
> Don't get me wrong, moving JDK 8u to git is doable, but it'll need some
> more thought than 11u.
> That's what I was thinking.
> 
> My preference would be to do it in steps:
> 1.) HG forest -> monorepo conversion (using JEP 296 scripts)
> 2.) HG -> git conversion using Skara tooling (should take care of
>  proper metadata, etc.)
> Yes, I think this is the right way to go about it. If nothing else
> because the tools for the second step assumed that the first one had
> happened.
> 
> The really tricky differences I see happen at step 1 (although they
> might have a knock-on effect at step 2)
> 
> I think converting 8u to a Git mono-repo might not be as complicated
> and risky as it seems. We have to remember that most of the work has
> already been done by JEP 296 in jdk10. That process already converted
> all the old history and as nobody seems to have been complaining about
> it in jdk10 and later, that conversion must have been fine. I.e. The
> jdk 10 repository naturally contains the sources of jdk8 from which is
> was initially forked (I think at jdk8-b120).
> 
> So we "only" have to take a clone of jdk10 or 11 up to the tag jdk8-b120:
> 
> hg clone -r jdk8-b120 jdk11u-dev/ jdk11u-dev-jdk8-b120
> 
> This will give us a mono-repo at the stage of jdk8-b120. Afterwards,
> we only have to use the JEP 296 scripts or something equivalent (I
> hope that the Oracle build group can assist or at least give some good
> advice here :)
> 
> From a quick look at the "jdk11u-dev-jdk8-b120" repo which I created
> with the above command, it looks like the "JEP 296" script worked as
> follows:
> - let's assume were at a changeset which merges all the subrepos at a
> specific build tag "jdk8-bXX" (builds were usually  tagged weekly, so
> every jdk8-bXX tag corresponds to a weekly version).
> - sequentially transplant all the changes from a single subrepo up
> until the next tag "jdk8-bXX+1" into the new repo
> - do this sequentially for all the other sub-repositories
> - create a merge change in the new mono-repo which merges all the
> newly transplanted changes and tag it with "jdk8-bXX+1"
> - verify that the sources of the new mono-repo at tag "jdk8-bXX+1" are
> bit-wise equal to the sources of the jdk8u forest when every single
> repo is synced to the same tag.
> - repeat the procedure for all the other jdk8-bXX tags
> 
> We can probably take the above jdk11u-dev-jdk8-b120 repo and follow
> the described procedure for all the additional changes between
> jdk8-b121 up to the latest jdk8u-XXX tag from the jdk8u-dev forest.
> Needless to say that the Corretto team is fully supporting the
> transition of jdk8u from a Mercurial forest to a Git mono-repo and
> will be happy to get involved into the process :)
> 
> Best regards,
> Volker
> 
> JEP 296 had to relocate all the java code into modules
> JEP 296 did not have to retain (and therefore relocate) all of the
> java code we need to keep in jdk8u (corba?, jaxb?, jaxws?)
> 
> I don't suppose these differences would be impossible to sort out but I
> think they would require some effort.
> 
> regards,
> 
> 
> Andrew Dinn
> -----------
>