Looking ahead: proposed Hg forest consolidation for JDK 10

[Andrew Hughes]

----- Original Message -----
> Hello,
> 
> Looking ahead to JDK 10, a group of JDK engineers have been exploring
> consolidating the large number of Hg repositories in an open JDK forest
> to a single one with the goal of using the consolidated arrangement for
> JDK 10.
> 
> This message is being sent to jdk9-dev since a jdk10-dev alias to
> discuss JDK 10 doesn't exist yet.
> 
> A JEP describing the project has been submitted :
> 
>      JDK-8167368: Consolidate JDK 10 OpenJDK repositories to a single
> repository
>      https://bugs.openjdk.java.net/browse/JDK-8167368
> 
> The text of the JEP describes the motivation and current state of the
> work in more detail, including proposed changes to the file layout.
> Publication of the prototype consolidated repository is planned, but not
> done yet. The email below has a list of additional anticipated questions
> and answers.
> 
> We feel this consolidated arrangement offers some significant structural
> advantages for managing the JDK's source code and we are now asking for
> feedback on this potential change. In particular, if you feel there is a
> show-stopper problem with making this change, please let us know!
> 
> I'd like to acknowledge the work of Stefan Sarne, Stuart Marks, and
> Ingemar Aberg participating in discussions leading up to the prototype
> and I'd like to especially recognize the contributions of Erik Helin for
> savvy Hg manipulations and Erik Joelsson for skillful build wrangling in
> this project.
> 
> Please send initial comments by October 18, 2016.
> 
> Cheers,
> 
> -Joe
> 
> Q: What about the set of forests for JDK 10? Are we going to have
> master, dev, client, hotspot, etc. the same set as in 9?
> A: That is a separate question from the repository consolidation, but
> there will likely be simplifications here too. Discussions on that point
> will come later.
> 
> Q: I usually just build the code in repo X today. Will I have have to
> build the *whole JDK* now?
> A: Not necessarily. The same top-level build targets should work in the
> consolidated forest.
> 
> Q: Does disk usage change?
> A: The total disk usage of the current forest compared to the
> consolidated forest is nearly the same.
> 
> Q: In more detail, how were the changesets imported?
> A: The scripts used for the consolidation conversion are attached to the
> JEP.
> 
> Q: What happens to the Hg hashes?
> A: The conversion scheme used in the prototype does *not* preserve Hg
> hashes of changesets compared the current forests. However, the bug ids
> are preserved and can be searched for. In addition, one or more
> pre-consolidation forests should be archived in perpetuity so that URLs
> in bug comments continue to work, etc.
> 
> A mapping of the old hashes to the corresponding new hashes might be
> generated and placed in the final new repo.
> 
> Q: I'm allergic to tabs; what about jcheck?
> A: If history is preserved, the checking done by jcheck needs to be
> modified for the consolidated forest. One way to do this is to augment
> the white lists used in jcheck with the conflicting changesets. This
> approach may not be elegant, but it is effective and doesn't appear to
> appreciably impact jcheck running times.
> 
> Q: Will the future 9 update forest also have this consolidation
> restructuring?
> A: The script used to do the consolidation conversion is deterministic
> and could be run to create the  9 update forest in the future at the
> discretion of the 9 update team.
> 
> Q: For backports for forwardports, will there be a script to translate
> patch files across the consolidation boundary?
> A: That work is planned, but not yet done; see JDK-8165623: Create patch
> translator to update paths pre/post consolidation.
> 
> Q: It's the 21st century and I develop using an IDE. That is still going
> to work, right?
> A: The prototype to date does include updating the various IDE support
> files, but bug JDK-8167142 has been filed to track that work.
> 
> 

As someone who regularly works with the whole set of repositories,
and has experienced the pain that was the forest extension back in the
day, I wholeheartedly welcome this change. For me, having to remember
to check out or update all those separate trees is just additional pain.
With the addition of yet another repo in OpenJDK 8 (nashorn), I thought
things were going in the opposite direction, so this comes as a welcome
surprise. While I can see the benefits of having HotSpot, langtools and
jdk in separate repos - they are largely separate pieces with separate
groups working on them - the reasoning to have JAXP, JAXWS and CORBA
separate always seemed bizarre to me.

I think this will also lower the headache for newcomers in getting hold
of OpenJDK. Having to download all these separate repositories is unusual
and confusing. Now if we only get rid of having both 7 and 7u, 8 and 8u, etc.
as well...

I'm curious as to how much the directory structure will change. The least
disruptive would be to just have hotspot, jaxws, jaxp, jdk, langtools, corba
and nashorn becomes regular subdirectories in the root repository, as this
wouldn't require any build changes. However, the bug suggests that the tree
structures will be merged. I hope this won't be as disruptive as the changes
in 9, and will instead largely keep the same structure, but merged together.
With 9, I find myself having to use find to locate where files have moved to,
and I haven't really seen any gain from that change.

I do think it's very important to retain as much history as possible. I appreciate
that the changeset IDs will change - that happens in backporting too - but
being able to trace individual changes to bug IDs is incredibly important.

One concern I do share with Goetz is the speed of the resulting repository.
The JDK repository is already pretty slow in operation. 

Is there an example converted repository available? I think this would answer
a lot of questions.

Incidentally, it may be helpful to leave this open for comments later than the
18th, as that's the date of the upcoming CPU.

Thanks,
-- 
Andrew :)

Senior Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net)
Fingerprint = 5132 579D D154 0ED2 3E04  C5A0 CFDA 0F9B 3596 4222