[11u] Proposal: Switch jdk11u development to Git/Skara with 11.0.13 cycle

[Vladimir Kempik]

Hello

We have observed one issue on jdk13u-dev after switching to skara: doing clean backport becomes a lot more complicated than with hg, as result it slowed our backporting pace.

Regards, Vladimir.

Volker Simonis <volker.simonis at gmail.com> 17 февраля 2021 г. 16:54:46 написал:

Hi Erik^2 :)

thank you so much for the detailed description. I'm especially excited
that you've already published fully consolidated mono-repos for jdk 6
to 9. That's really very much appreciated!

Now that the heavy lifting has been done, I hope we can seriously
consider migrating 8u to Git as well. @OpenJDK8u-Maintainers - what
blockers are you still seeing? Is there anything the community can
help with to speedup the migration?

Thank you and best regards,
Volker

On Tue, Feb 16, 2021 at 8:31 PM <erik.joelsson at oracle.com> wrote:

Hello,

As you may have seen, Erik Duveblad just posted about our prototype
conversions on skara-dev [1]. These are based on the same kind of
forest->mono repo conversion, followed by hg->git conversion that
mainline went through.

I have been working for a while now on adapting the scripts from JEP 296
to work for the old update releases. This has required more work than I
first anticipated as the problem now requires a more generalized
solution, but I believe I now have something that works reasonably well.
If you are curious on the details, please continue reading.

Volker's description of the processes is pretty accurate on a high
level. The main difference in 8u compared to a main release is that the
tags are not sequential in history. This means that we can't transplant
(or splice as it's called in Mercurial terms) changes as freely on top
of each tagged change. In the original JDK 10 consolidation, we had a
handful of changes that weren't linear (around the transition between
JDK 9 and 10), and I handled them differently (basically just merging
them together to recreate the point in history, but not splicing
anything on top afterwards). Since they were only a handful, I could
just list them out explicitly in the script. For 8u, a majority of the
tags are non linear, so a big problem was creating an algorithm that
would automatically figure out a reasonable line of tags in sequence
that I could use to splice changes on top of.

Another big problem is that there are often tags merged together in
different order in different repos. When this happens, the script needs
to figure out the best order to do the conversion. I have tried to
automate this process as much as possible. The ordering is sensitive
when some of the tags are used to splice changes on top of them. There
are also cases of tags missing completely.

In the end, the state of the mono repo at each tag is verified to be
exactly the same as the forest of repos at each tag. The state of
changes between tags is however less predictable in 8u compared to
mainline due to the large amount of non linear tags.

For the extra curious, here is what the current algorithm for choosing
the "splice tags" looks like:

1. Start with the first tag

2. Continue with each consecutive build for current release until a gap
in build numbers is detected (we typically have update release builds
b30 that probably aren't that interesting)

3. Find the next release in order and then find the first tag in that
release that is also a descendant of the previous splice tag in every repo.

4. Repeat from 2 until all tags have been considered.

/Erik

[1]
https://mail.openjdk.java.net/pipermail/skara-dev/2021-February/004228.html

On 2021-02-12 13:33, Volker Simonis wrote:
On Fri, Feb 12, 2021 at 6:40 PM Andrew Dinn <adinn at redhat.com> wrote:
On 12/02/2021 16:37, Severin Gehwolf wrote:
Don't get me wrong, moving JDK 8u to git is doable, but it'll need some
more thought than 11u.
That's what I was thinking.

My preference would be to do it in steps:
1.) HG forest -> monorepo conversion (using JEP 296 scripts)
2.) HG -> git conversion using Skara tooling (should take care of
  proper metadata, etc.)
Yes, I think this is the right way to go about it. If nothing else
because the tools for the second step assumed that the first one had
happened.

The really tricky differences I see happen at step 1 (although they
might have a knock-on effect at step 2)

I think converting 8u to a Git mono-repo might not be as complicated
and risky as it seems. We have to remember that most of the work has
already been done by JEP 296 in jdk10. That process already converted
all the old history and as nobody seems to have been complaining about
it in jdk10 and later, that conversion must have been fine. I.e. The
jdk 10 repository naturally contains the sources of jdk8 from which is
was initially forked (I think at jdk8-b120).

So we "only" have to take a clone of jdk10 or 11 up to the tag jdk8-b120:

hg clone -r jdk8-b120 jdk11u-dev/ jdk11u-dev-jdk8-b120

This will give us a mono-repo at the stage of jdk8-b120. Afterwards,
we only have to use the JEP 296 scripts or something equivalent (I
hope that the Oracle build group can assist or at least give some good
advice here :)

>From a quick look at the "jdk11u-dev-jdk8-b120" repo which I created
with the above command, it looks like the "JEP 296" script worked as
follows:
- let's assume were at a changeset which merges all the subrepos at a
specific build tag "jdk8-bXX" (builds were usually  tagged weekly, so
every jdk8-bXX tag corresponds to a weekly version).
- sequentially transplant all the changes from a single subrepo up
until the next tag "jdk8-bXX+1" into the new repo
- do this sequentially for all the other sub-repositories
- create a merge change in the new mono-repo which merges all the
newly transplanted changes and tag it with "jdk8-bXX+1"
- verify that the sources of the new mono-repo at tag "jdk8-bXX+1" are
bit-wise equal to the sources of the jdk8u forest when every single
repo is synced to the same tag.
- repeat the procedure for all the other jdk8-bXX tags

We can probably take the above jdk11u-dev-jdk8-b120 repo and follow
the described procedure for all the additional changes between
jdk8-b121 up to the latest jdk8u-XXX tag from the jdk8u-dev forest.
Needless to say that the Corretto team is fully supporting the
transition of jdk8u from a Mercurial forest to a Git mono-repo and
will be happy to get involved into the process :)

Best regards,
Volker

JEP 296 had to relocate all the java code into modules
JEP 296 did not have to retain (and therefore relocate) all of the
java code we need to keep in jdk8u (corba?, jaxb?, jaxws?)

I don't suppose these differences would be impossible to sort out but I
think they would require some effort.

regards,


Andrew Dinn
-----------