[11u] Proposal: Switch jdk11u development to Git/Skara with 11.0.13 cycle

[Lindenmaier, Goetz]

Hi,

I think it is about time to start a thread of its own to discuss this for 8.
What do you think?

Best regards,
  Goetz.

> -----Original Message-----
> From: jdk8u-dev <jdk8u-dev-retn at openjdk.java.net> On Behalf Of Volker
> Simonis
> Sent: Wednesday, February 17, 2021 2:54 PM
> To: Erik Joelsson <erik.joelsson at oracle.com>; Erik Helin
> <erik.helin at oracle.com>
> Cc: jdk-updates-dev <jdk-updates-dev at openjdk.java.net>; jdk8u-dev <jdk8u-
> dev at openjdk.java.net>
> Subject: Re: [11u] Proposal: Switch jdk11u development to Git/Skara with
> 11.0.13 cycle
> 
> Hi Erik^2 :)
> 
> thank you so much for the detailed description. I'm especially excited
> that you've already published fully consolidated mono-repos for jdk 6
> to 9. That's really very much appreciated!
> 
> Now that the heavy lifting has been done, I hope we can seriously
> consider migrating 8u to Git as well. @OpenJDK8u-Maintainers - what
> blockers are you still seeing? Is there anything the community can
> help with to speedup the migration?
> 
> Thank you and best regards,
> Volker
> 
> On Tue, Feb 16, 2021 at 8:31 PM <erik.joelsson at oracle.com> wrote:
> >
> > Hello,
> >
> > As you may have seen, Erik Duveblad just posted about our prototype
> > conversions on skara-dev [1]. These are based on the same kind of
> > forest->mono repo conversion, followed by hg->git conversion that
> > mainline went through.
> >
> > I have been working for a while now on adapting the scripts from JEP 296
> > to work for the old update releases. This has required more work than I
> > first anticipated as the problem now requires a more generalized
> > solution, but I believe I now have something that works reasonably well.
> > If you are curious on the details, please continue reading.
> >
> > Volker's description of the processes is pretty accurate on a high
> > level. The main difference in 8u compared to a main release is that the
> > tags are not sequential in history. This means that we can't transplant
> > (or splice as it's called in Mercurial terms) changes as freely on top
> > of each tagged change. In the original JDK 10 consolidation, we had a
> > handful of changes that weren't linear (around the transition between
> > JDK 9 and 10), and I handled them differently (basically just merging
> > them together to recreate the point in history, but not splicing
> > anything on top afterwards). Since they were only a handful, I could
> > just list them out explicitly in the script. For 8u, a majority of the
> > tags are non linear, so a big problem was creating an algorithm that
> > would automatically figure out a reasonable line of tags in sequence
> > that I could use to splice changes on top of.
> >
> > Another big problem is that there are often tags merged together in
> > different order in different repos. When this happens, the script needs
> > to figure out the best order to do the conversion. I have tried to
> > automate this process as much as possible. The ordering is sensitive
> > when some of the tags are used to splice changes on top of them. There
> > are also cases of tags missing completely.
> >
> > In the end, the state of the mono repo at each tag is verified to be
> > exactly the same as the forest of repos at each tag. The state of
> > changes between tags is however less predictable in 8u compared to
> > mainline due to the large amount of non linear tags.
> >
> > For the extra curious, here is what the current algorithm for choosing
> > the "splice tags" looks like:
> >
> > 1. Start with the first tag
> >
> > 2. Continue with each consecutive build for current release until a gap
> > in build numbers is detected (we typically have update release builds
> >  >b30 that probably aren't that interesting)
> >
> > 3. Find the next release in order and then find the first tag in that
> > release that is also a descendant of the previous splice tag in every repo.
> >
> > 4. Repeat from 2 until all tags have been considered.
> >
> > /Erik
> >
> > [1]
> > https://mail.openjdk.java.net/pipermail/skara-dev/2021-
> February/004228.html
> >
> > On 2021-02-12 13:33, Volker Simonis wrote:
> > > On Fri, Feb 12, 2021 at 6:40 PM Andrew Dinn <adinn at redhat.com> wrote:
> > >> On 12/02/2021 16:37, Severin Gehwolf wrote:
> > >>> Don't get me wrong, moving JDK 8u to git is doable, but it'll need some
> > >>> more thought than 11u.
> > >> That's what I was thinking.
> > >>
> > >>> My preference would be to do it in steps:
> > >>> 1.) HG forest -> monorepo conversion (using JEP 296 scripts)
> > >>> 2.) HG -> git conversion using Skara tooling (should take care of
> > >>>       proper metadata, etc.)
> > >> Yes, I think this is the right way to go about it. If nothing else
> > >> because the tools for the second step assumed that the first one had
> > >> happened.
> > >>
> > >> The really tricky differences I see happen at step 1 (although they
> > >> might have a knock-on effect at step 2)
> > >>
> > > I think converting 8u to a Git mono-repo might not be as complicated
> > > and risky as it seems. We have to remember that most of the work has
> > > already been done by JEP 296 in jdk10. That process already converted
> > > all the old history and as nobody seems to have been complaining about
> > > it in jdk10 and later, that conversion must have been fine. I.e. The
> > > jdk 10 repository naturally contains the sources of jdk8 from which is
> > > was initially forked (I think at jdk8-b120).
> > >
> > > So we "only" have to take a clone of jdk10 or 11 up to the tag jdk8-b120:
> > >
> > > hg clone -r jdk8-b120 jdk11u-dev/ jdk11u-dev-jdk8-b120
> > >
> > > This will give us a mono-repo at the stage of jdk8-b120. Afterwards,
> > > we only have to use the JEP 296 scripts or something equivalent (I
> > > hope that the Oracle build group can assist or at least give some good
> > > advice here :)
> > >
> > >  From a quick look at the "jdk11u-dev-jdk8-b120" repo which I created
> > > with the above command, it looks like the "JEP 296" script worked as
> > > follows:
> > > - let's assume were at a changeset which merges all the subrepos at a
> > > specific build tag "jdk8-bXX" (builds were usually  tagged weekly, so
> > > every jdk8-bXX tag corresponds to a weekly version).
> > > - sequentially transplant all the changes from a single subrepo up
> > > until the next tag "jdk8-bXX+1" into the new repo
> > > - do this sequentially for all the other sub-repositories
> > > - create a merge change in the new mono-repo which merges all the
> > > newly transplanted changes and tag it with "jdk8-bXX+1"
> > > - verify that the sources of the new mono-repo at tag "jdk8-bXX+1" are
> > > bit-wise equal to the sources of the jdk8u forest when every single
> > > repo is synced to the same tag.
> > > - repeat the procedure for all the other jdk8-bXX tags
> > >
> > > We can probably take the above jdk11u-dev-jdk8-b120 repo and follow
> > > the described procedure for all the additional changes between
> > > jdk8-b121 up to the latest jdk8u-XXX tag from the jdk8u-dev forest.
> > > Needless to say that the Corretto team is fully supporting the
> > > transition of jdk8u from a Mercurial forest to a Git mono-repo and
> > > will be happy to get involved into the process :)
> > >
> > > Best regards,
> > > Volker
> > >
> > >>     JEP 296 had to relocate all the java code into modules
> > >>     JEP 296 did not have to retain (and therefore relocate) all of the
> > >> java code we need to keep in jdk8u (corba?, jaxb?, jaxws?)
> > >>
> > >> I don't suppose these differences would be impossible to sort out but I
> > >> think they would require some effort.
> > >>
> > >> regards,
> > >>
> > >>
> > >> Andrew Dinn
> > >> -----------
> > >>