[11u] Proposal: Switch jdk11u development to Git/Skara with 11.0.13 cycle

[erik.joelsson at oracle.com]

(resending this with full to/cc list)

Hello,

As you may have seen, Erik Duveblad just posted about our prototype 
conversions on skara-dev [1]. These are based on the same kind of 
forest->mono repo conversion, followed by hg->git conversion that 
mainline went through.

I have been working for a while now on adapting the scripts from JEP 296 
to work for the old update releases. This has required more work than I 
first anticipated as the problem now requires a more generalized 
solution, but I believe I now have something that works reasonably well. 
If you are curious on the details, please continue reading.

Volker's description of the processes is pretty accurate on a high 
level. The main difference in 8u compared to a main release is that the 
tags are not sequential in history. This means that we can't transplant 
(or splice as it's called in Mercurial terms) changes as freely on top 
of each tagged change. In the original JDK 10 consolidation, we had a 
handful of changes that weren't linear (around the transition between 
JDK 9 and 10), and I handled them differently (basically just merging 
them together to recreate the point in history, but not splicing 
anything on top afterwards). Since they were only a handful, I could 
just list them out explicitly in the script. For 8u, a majority of the 
tags are non linear, so a big problem was creating an algorithm that 
would automatically figure out a reasonable line of tags in sequence 
that I could use to splice changes on top of.

Another big problem is that there are often tags merged together in 
different order in different repos. When this happens, the script needs 
to figure out the best order to do the conversion. I have tried to 
automate this process as much as possible. The ordering is sensitive 
when some of the tags are used to splice changes on top of them. There 
are also cases of tags missing completely.

In the end, the state of the mono repo at each tag is verified to be 
exactly the same as the forest of repos at each tag. The state of 
changes between tags is however less predictable in 8u compared to 
mainline due to the large amount of non linear tags.

For the extra curious, here is what the current algorithm for choosing 
the "splice tags" looks like:

1. Start with the first tag

2. Continue with each consecutive build for current release until a gap 
in build numbers is detected (we typically have update release builds 
 >b30 that probably aren't that interesting)

3. Find the next release in order and then find the first tag in that 
release that is also a descendant of the previous splice tag in every repo.

4. Repeat from 2 until all tags have been considered.

/Erik

[1] 
https://mail.openjdk.java.net/pipermail/skara-dev/2021-February/004228.html

On 2021-02-12 13:33, Volker Simonis wrote:
> On Fri, Feb 12, 2021 at 6:40 PM Andrew Dinn <adinn at redhat.com> wrote:
>> On 12/02/2021 16:37, Severin Gehwolf wrote:
>>> Don't get me wrong, moving JDK 8u to git is doable, but it'll need some
>>> more thought than 11u.
>> That's what I was thinking.
>>
>>> My preference would be to do it in steps:
>>> 1.) HG forest -> monorepo conversion (using JEP 296 scripts)
>>> 2.) HG -> git conversion using Skara tooling (should take care of
>>>       proper metadata, etc.)
>> Yes, I think this is the right way to go about it. If nothing else
>> because the tools for the second step assumed that the first one had
>> happened.
>>
>> The really tricky differences I see happen at step 1 (although they
>> might have a knock-on effect at step 2)
>>
> I think converting 8u to a Git mono-repo might not be as complicated
> and risky as it seems. We have to remember that most of the work has
> already been done by JEP 296 in jdk10. That process already converted
> all the old history and as nobody seems to have been complaining about
> it in jdk10 and later, that conversion must have been fine. I.e. The
> jdk 10 repository naturally contains the sources of jdk8 from which is
> was initially forked (I think at jdk8-b120).
>
> So we "only" have to take a clone of jdk10 or 11 up to the tag jdk8-b120:
>
> hg clone -r jdk8-b120 jdk11u-dev/ jdk11u-dev-jdk8-b120
>
> This will give us a mono-repo at the stage of jdk8-b120. Afterwards,
> we only have to use the JEP 296 scripts or something equivalent (I
> hope that the Oracle build group can assist or at least give some good
> advice here :)
>
>  From a quick look at the "jdk11u-dev-jdk8-b120" repo which I created
> with the above command, it looks like the "JEP 296" script worked as
> follows:
> - let's assume were at a changeset which merges all the subrepos at a
> specific build tag "jdk8-bXX" (builds were usually  tagged weekly, so
> every jdk8-bXX tag corresponds to a weekly version).
> - sequentially transplant all the changes from a single subrepo up
> until the next tag "jdk8-bXX+1" into the new repo
> - do this sequentially for all the other sub-repositories
> - create a merge change in the new mono-repo which merges all the
> newly transplanted changes and tag it with "jdk8-bXX+1"
> - verify that the sources of the new mono-repo at tag "jdk8-bXX+1" are
> bit-wise equal to the sources of the jdk8u forest when every single
> repo is synced to the same tag.
> - repeat the procedure for all the other jdk8-bXX tags
>
> We can probably take the above jdk11u-dev-jdk8-b120 repo and follow
> the described procedure for all the additional changes between
> jdk8-b121 up to the latest jdk8u-XXX tag from the jdk8u-dev forest.
> Needless to say that the Corretto team is fully supporting the
> transition of jdk8u from a Mercurial forest to a Git mono-repo and
> will be happy to get involved into the process :)
>
> Best regards,
> Volker
>
>>     JEP 296 had to relocate all the java code into modules
>>     JEP 296 did not have to retain (and therefore relocate) all of the
>> java code we need to keep in jdk8u (corba?, jaxb?, jaxws?)
>>
>> I don't suppose these differences would be impossible to sort out but I
>> think they would require some effort.
>>
>> regards,
>>
>>
>> Andrew Dinn
>> -----------
>>