[11u] Proposal: Switch jdk11u development to Git/Skara with 11.0.13 cycle
erik.joelsson at oracle.com
erik.joelsson at oracle.com
Tue Feb 16 19:30:41 UTC 2021
Hello,
As you may have seen, Erik Duveblad just posted about our prototype
conversions on skara-dev [1]. These are based on the same kind of
forest->mono repo conversion, followed by hg->git conversion that
mainline went through.
I have been working for a while now on adapting the scripts from JEP 296
to work for the old update releases. This has required more work than I
first anticipated as the problem now requires a more generalized
solution, but I believe I now have something that works reasonably well.
If you are curious on the details, please continue reading.
Volker's description of the processes is pretty accurate on a high
level. The main difference in 8u compared to a main release is that the
tags are not sequential in history. This means that we can't transplant
(or splice as it's called in Mercurial terms) changes as freely on top
of each tagged change. In the original JDK 10 consolidation, we had a
handful of changes that weren't linear (around the transition between
JDK 9 and 10), and I handled them differently (basically just merging
them together to recreate the point in history, but not splicing
anything on top afterwards). Since they were only a handful, I could
just list them out explicitly in the script. For 8u, a majority of the
tags are non linear, so a big problem was creating an algorithm that
would automatically figure out a reasonable line of tags in sequence
that I could use to splice changes on top of.
Another big problem is that there are often tags merged together in
different order in different repos. When this happens, the script needs
to figure out the best order to do the conversion. I have tried to
automate this process as much as possible. The ordering is sensitive
when some of the tags are used to splice changes on top of them. There
are also cases of tags missing completely.
In the end, the state of the mono repo at each tag is verified to be
exactly the same as the forest of repos at each tag. The state of
changes between tags is however less predictable in 8u compared to
mainline due to the large amount of non linear tags.
For the extra curious, here is what the current algorithm for choosing
the "splice tags" looks like:
1. Start with the first tag
2. Continue with each consecutive build for current release until a gap
in build numbers is detected (we typically have update release builds
>b30 that probably aren't that interesting)
3. Find the next release in order and then find the first tag in that
release that is also a descendant of the previous splice tag in every repo.
4. Repeat from 2 until all tags have been considered.
/Erik
[1]
https://mail.openjdk.java.net/pipermail/skara-dev/2021-February/004228.html
On 2021-02-12 13:33, Volker Simonis wrote:
> On Fri, Feb 12, 2021 at 6:40 PM Andrew Dinn <adinn at redhat.com> wrote:
>> On 12/02/2021 16:37, Severin Gehwolf wrote:
>>> Don't get me wrong, moving JDK 8u to git is doable, but it'll need some
>>> more thought than 11u.
>> That's what I was thinking.
>>
>>> My preference would be to do it in steps:
>>> 1.) HG forest -> monorepo conversion (using JEP 296 scripts)
>>> 2.) HG -> git conversion using Skara tooling (should take care of
>>> proper metadata, etc.)
>> Yes, I think this is the right way to go about it. If nothing else
>> because the tools for the second step assumed that the first one had
>> happened.
>>
>> The really tricky differences I see happen at step 1 (although they
>> might have a knock-on effect at step 2)
>>
> I think converting 8u to a Git mono-repo might not be as complicated
> and risky as it seems. We have to remember that most of the work has
> already been done by JEP 296 in jdk10. That process already converted
> all the old history and as nobody seems to have been complaining about
> it in jdk10 and later, that conversion must have been fine. I.e. The
> jdk 10 repository naturally contains the sources of jdk8 from which is
> was initially forked (I think at jdk8-b120).
>
> So we "only" have to take a clone of jdk10 or 11 up to the tag jdk8-b120:
>
> hg clone -r jdk8-b120 jdk11u-dev/ jdk11u-dev-jdk8-b120
>
> This will give us a mono-repo at the stage of jdk8-b120. Afterwards,
> we only have to use the JEP 296 scripts or something equivalent (I
> hope that the Oracle build group can assist or at least give some good
> advice here :)
>
> From a quick look at the "jdk11u-dev-jdk8-b120" repo which I created
> with the above command, it looks like the "JEP 296" script worked as
> follows:
> - let's assume were at a changeset which merges all the subrepos at a
> specific build tag "jdk8-bXX" (builds were usually tagged weekly, so
> every jdk8-bXX tag corresponds to a weekly version).
> - sequentially transplant all the changes from a single subrepo up
> until the next tag "jdk8-bXX+1" into the new repo
> - do this sequentially for all the other sub-repositories
> - create a merge change in the new mono-repo which merges all the
> newly transplanted changes and tag it with "jdk8-bXX+1"
> - verify that the sources of the new mono-repo at tag "jdk8-bXX+1" are
> bit-wise equal to the sources of the jdk8u forest when every single
> repo is synced to the same tag.
> - repeat the procedure for all the other jdk8-bXX tags
>
> We can probably take the above jdk11u-dev-jdk8-b120 repo and follow
> the described procedure for all the additional changes between
> jdk8-b121 up to the latest jdk8u-XXX tag from the jdk8u-dev forest.
> Needless to say that the Corretto team is fully supporting the
> transition of jdk8u from a Mercurial forest to a Git mono-repo and
> will be happy to get involved into the process :)
>
> Best regards,
> Volker
>
>> JEP 296 had to relocate all the java code into modules
>> JEP 296 did not have to retain (and therefore relocate) all of the
>> java code we need to keep in jdk8u (corba?, jaxb?, jaxws?)
>>
>> I don't suppose these differences would be impossible to sort out but I
>> think they would require some effort.
>>
>> regards,
>>
>>
>> Andrew Dinn
>> -----------
>>
More information about the jdk-updates-dev
mailing list