Looking ahead: proposed Hg forest consolidation for JDK 10

[joe darcy]

Hello,

Looking ahead to JDK 10, a group of JDK engineers have been exploring 
consolidating the large number of Hg repositories in an open JDK forest 
to a single one with the goal of using the consolidated arrangement for 
JDK 10.

This message is being sent to jdk9-dev since a jdk10-dev alias to 
discuss JDK 10 doesn't exist yet.

A JEP describing the project has been submitted :

     JDK-8167368: Consolidate JDK 10 OpenJDK repositories to a single 
repository
     https://bugs.openjdk.java.net/browse/JDK-8167368

The text of the JEP describes the motivation and current state of the 
work in more detail, including proposed changes to the file layout. 
Publication of the prototype consolidated repository is planned, but not 
done yet. The email below has a list of additional anticipated questions 
and answers.

We feel this consolidated arrangement offers some significant structural 
advantages for managing the JDK's source code and we are now asking for 
feedback on this potential change. In particular, if you feel there is a 
show-stopper problem with making this change, please let us know!

I'd like to acknowledge the work of Stefan Sarne, Stuart Marks, and 
Ingemar Aberg participating in discussions leading up to the prototype 
and I'd like to especially recognize the contributions of Erik Helin for 
savvy Hg manipulations and Erik Joelsson for skillful build wrangling in 
this project.

Please send initial comments by October 18, 2016.

Cheers,

-Joe

Q: What about the set of forests for JDK 10? Are we going to have 
master, dev, client, hotspot, etc. the same set as in 9?
A: That is a separate question from the repository consolidation, but 
there will likely be simplifications here too. Discussions on that point 
will come later.

Q: I usually just build the code in repo X today. Will I have have to 
build the *whole JDK* now?
A: Not necessarily. The same top-level build targets should work in the 
consolidated forest.

Q: Does disk usage change?
A: The total disk usage of the current forest compared to the 
consolidated forest is nearly the same.

Q: In more detail, how were the changesets imported?
A: The scripts used for the consolidation conversion are attached to the 
JEP.

Q: What happens to the Hg hashes?
A: The conversion scheme used in the prototype does *not* preserve Hg 
hashes of changesets compared the current forests. However, the bug ids 
are preserved and can be searched for. In addition, one or more 
pre-consolidation forests should be archived in perpetuity so that URLs 
in bug comments continue to work, etc.

A mapping of the old hashes to the corresponding new hashes might be 
generated and placed in the final new repo.

Q: I'm allergic to tabs; what about jcheck?
A: If history is preserved, the checking done by jcheck needs to be 
modified for the consolidated forest. One way to do this is to augment 
the white lists used in jcheck with the conflicting changesets. This 
approach may not be elegant, but it is effective and doesn't appear to 
appreciably impact jcheck running times.

Q: Will the future 9 update forest also have this consolidation 
restructuring?
A: The script used to do the consolidation conversion is deterministic 
and could be run to create the  9 update forest in the future at the 
discretion of the 9 update team.

Q: For backports for forwardports, will there be a script to translate 
patch files across the consolidation boundary?
A: That work is planned, but not yet done; see JDK-8165623: Create patch 
translator to update paths pre/post consolidation.

Q: It's the 21st century and I develop using an IDE. That is still going 
to work, right?
A: The prototype to date does include updating the various IDE support 
files, but bug JDK-8167142 has been filed to track that work.