[11u] Proposal: Switch jdk11u development to Git/Skara with 11.0.13 cycle

Wed Feb 17 13:54:07 UTC 2021

Hi Erik^2 :)

thank you so much for the detailed description. I'm especially excited
that you've already published fully consolidated mono-repos for jdk 6
to 9. That's really very much appreciated!

Now that the heavy lifting has been done, I hope we can seriously
consider migrating 8u to Git as well. @OpenJDK8u-Maintainers - what
blockers are you still seeing? Is there anything the community can
help with to speedup the migration?

Thank you and best regards,
Volker

On Tue, Feb 16, 2021 at 8:31 PM <erik.joelsson at oracle.com> wrote:
>
> Hello,
>
> As you may have seen, Erik Duveblad just posted about our prototype
> conversions on skara-dev [1]. These are based on the same kind of
> forest->mono repo conversion, followed by hg->git conversion that
> mainline went through.
>
> I have been working for a while now on adapting the scripts from JEP 296
> to work for the old update releases. This has required more work than I
> first anticipated as the problem now requires a more generalized
> solution, but I believe I now have something that works reasonably well.
> If you are curious on the details, please continue reading.
>
> Volker's description of the processes is pretty accurate on a high
> level. The main difference in 8u compared to a main release is that the
> tags are not sequential in history. This means that we can't transplant
> (or splice as it's called in Mercurial terms) changes as freely on top
> of each tagged change. In the original JDK 10 consolidation, we had a
> handful of changes that weren't linear (around the transition between
> JDK 9 and 10), and I handled them differently (basically just merging
> them together to recreate the point in history, but not splicing
> anything on top afterwards). Since they were only a handful, I could
> just list them out explicitly in the script. For 8u, a majority of the
> tags are non linear, so a big problem was creating an algorithm that
> would automatically figure out a reasonable line of tags in sequence
> that I could use to splice changes on top of.
>
> Another big problem is that there are often tags merged together in
> different order in different repos. When this happens, the script needs
> to figure out the best order to do the conversion. I have tried to
> automate this process as much as possible. The ordering is sensitive
> when some of the tags are used to splice changes on top of them. There
> are also cases of tags missing completely.
>
> In the end, the state of the mono repo at each tag is verified to be
> exactly the same as the forest of repos at each tag. The state of
> changes between tags is however less predictable in 8u compared to
> mainline due to the large amount of non linear tags.
>
> For the extra curious, here is what the current algorithm for choosing
> the "splice tags" looks like:
>
> 1. Start with the first tag
>
> 2. Continue with each consecutive build for current release until a gap
> in build numbers is detected (we typically have update release builds
>  >b30 that probably aren't that interesting)
>
> 3. Find the next release in order and then find the first tag in that
> release that is also a descendant of the previous splice tag in every repo.
>
> 4. Repeat from 2 until all tags have been considered.
>
> /Erik
>
> [1]
> https://mail.openjdk.java.net/pipermail/skara-dev/2021-February/004228.html
>
> On 2021-02-12 13:33, Volker Simonis wrote:
> > On Fri, Feb 12, 2021 at 6:40 PM Andrew Dinn <adinn at redhat.com> wrote:
> >> On 12/02/2021 16:37, Severin Gehwolf wrote:
> >>> Don't get me wrong, moving JDK 8u to git is doable, but it'll need some
> >>> more thought than 11u.
> >> That's what I was thinking.
> >>
> >>> My preference would be to do it in steps:
> >>> 1.) HG forest -> monorepo conversion (using JEP 296 scripts)
> >>> 2.) HG -> git conversion using Skara tooling (should take care of
> >>>       proper metadata, etc.)
> >> Yes, I think this is the right way to go about it. If nothing else
> >> because the tools for the second step assumed that the first one had
> >> happened.
> >>
> >> The really tricky differences I see happen at step 1 (although they
> >> might have a knock-on effect at step 2)
> >>
> > I think converting 8u to a Git mono-repo might not be as complicated
> > and risky as it seems. We have to remember that most of the work has
> > already been done by JEP 296 in jdk10. That process already converted
> > all the old history and as nobody seems to have been complaining about
> > it in jdk10 and later, that conversion must have been fine. I.e. The
> > jdk 10 repository naturally contains the sources of jdk8 from which is
> > was initially forked (I think at jdk8-b120).
> >
> > So we "only" have to take a clone of jdk10 or 11 up to the tag jdk8-b120:
> >
> > hg clone -r jdk8-b120 jdk11u-dev/ jdk11u-dev-jdk8-b120
> >
> > This will give us a mono-repo at the stage of jdk8-b120. Afterwards,
> > we only have to use the JEP 296 scripts or something equivalent (I
> > hope that the Oracle build group can assist or at least give some good
> > advice here :)
> >
> >  From a quick look at the "jdk11u-dev-jdk8-b120" repo which I created
> > with the above command, it looks like the "JEP 296" script worked as
> > follows:
> > - let's assume were at a changeset which merges all the subrepos at a
> > specific build tag "jdk8-bXX" (builds were usually  tagged weekly, so
> > every jdk8-bXX tag corresponds to a weekly version).
> > - sequentially transplant all the changes from a single subrepo up
> > until the next tag "jdk8-bXX+1" into the new repo
> > - do this sequentially for all the other sub-repositories
> > - create a merge change in the new mono-repo which merges all the
> > newly transplanted changes and tag it with "jdk8-bXX+1"
> > - verify that the sources of the new mono-repo at tag "jdk8-bXX+1" are
> > bit-wise equal to the sources of the jdk8u forest when every single
> > repo is synced to the same tag.
> > - repeat the procedure for all the other jdk8-bXX tags
> >
> > We can probably take the above jdk11u-dev-jdk8-b120 repo and follow
> > the described procedure for all the additional changes between
> > jdk8-b121 up to the latest jdk8u-XXX tag from the jdk8u-dev forest.
> > Needless to say that the Corretto team is fully supporting the
> > transition of jdk8u from a Mercurial forest to a Git mono-repo and
> > will be happy to get involved into the process :)
> >
> > Best regards,
> > Volker
> >
> >>     JEP 296 had to relocate all the java code into modules
> >>     JEP 296 did not have to retain (and therefore relocate) all of the
> >> java code we need to keep in jdk8u (corba?, jaxb?, jaxws?)
> >>
> >> I don't suppose these differences would be impossible to sort out but I
> >> think they would require some effort.
> >>
> >> regards,
> >>
> >>
> >> Andrew Dinn
> >> -----------
> >>