Looking ahead: proposed Hg forest consolidation for JDK 10

[Stephen Colebourne]

On those few occasions where I have worked with the main OpenJDK
repos, it has been painful. But this wasn't because of the separation
(which the seemed to be handled fine in the initial setup phase where
the scripts clone everything), but because of the performance (hg
seems dog-slow to me compared with git, but perhaps it is just a large
repo).

I note that the linux repo has a split between the current repo and a
historic repo, that can be linked using git grafts:
http://stackoverflow.com/questions/3264283/linux-kernel-historical-git-repository-with-full-history/8130239#8130239
I wonder if there is a similar concept for hg that would allow the
repo to be split, removing older history while still allowing it to be
pieced together.

On the feedback that hotspot is perhaps more separate than the rest, I
tend to agree. While there are clearly times where the two projects
need to be updated together, hopefully this is regularly rare. I would
have thought that the two teams could both work on a branch until the
flag day, when it would then just be a simple merge on each repo
(speaking it git terminology, hg terminology may differ).

In summary, my concern is primarily performance. Its already slow, and
this change can only make it slower.

Stephen


On 11 October 2016 at 03:11, joe darcy <joe.darcy at oracle.com> wrote:
> Hello,
>
> Looking ahead to JDK 10, a group of JDK engineers have been exploring
> consolidating the large number of Hg repositories in an open JDK forest to a
> single one with the goal of using the consolidated arrangement for JDK 10.
>
> This message is being sent to jdk9-dev since a jdk10-dev alias to discuss
> JDK 10 doesn't exist yet.
>
> A JEP describing the project has been submitted :
>
>     JDK-8167368: Consolidate JDK 10 OpenJDK repositories to a single
> repository
>     https://bugs.openjdk.java.net/browse/JDK-8167368
>
> The text of the JEP describes the motivation and current state of the work
> in more detail, including proposed changes to the file layout. Publication
> of the prototype consolidated repository is planned, but not done yet. The
> email below has a list of additional anticipated questions and answers.
>
> We feel this consolidated arrangement offers some significant structural
> advantages for managing the JDK's source code and we are now asking for
> feedback on this potential change. In particular, if you feel there is a
> show-stopper problem with making this change, please let us know!
>
> I'd like to acknowledge the work of Stefan Sarne, Stuart Marks, and Ingemar
> Aberg participating in discussions leading up to the prototype and I'd like
> to especially recognize the contributions of Erik Helin for savvy Hg
> manipulations and Erik Joelsson for skillful build wrangling in this
> project.
>
> Please send initial comments by October 18, 2016.
>
> Cheers,
>
> -Joe
>
> Q: What about the set of forests for JDK 10? Are we going to have master,
> dev, client, hotspot, etc. the same set as in 9?
> A: That is a separate question from the repository consolidation, but there
> will likely be simplifications here too. Discussions on that point will come
> later.
>
> Q: I usually just build the code in repo X today. Will I have have to build
> the *whole JDK* now?
> A: Not necessarily. The same top-level build targets should work in the
> consolidated forest.
>
> Q: Does disk usage change?
> A: The total disk usage of the current forest compared to the consolidated
> forest is nearly the same.
>
> Q: In more detail, how were the changesets imported?
> A: The scripts used for the consolidation conversion are attached to the
> JEP.
>
> Q: What happens to the Hg hashes?
> A: The conversion scheme used in the prototype does *not* preserve Hg hashes
> of changesets compared the current forests. However, the bug ids are
> preserved and can be searched for. In addition, one or more
> pre-consolidation forests should be archived in perpetuity so that URLs in
> bug comments continue to work, etc.
>
> A mapping of the old hashes to the corresponding new hashes might be
> generated and placed in the final new repo.
>
> Q: I'm allergic to tabs; what about jcheck?
> A: If history is preserved, the checking done by jcheck needs to be modified
> for the consolidated forest. One way to do this is to augment the white
> lists used in jcheck with the conflicting changesets. This approach may not
> be elegant, but it is effective and doesn't appear to appreciably impact
> jcheck running times.
>
> Q: Will the future 9 update forest also have this consolidation
> restructuring?
> A: The script used to do the consolidation conversion is deterministic and
> could be run to create the  9 update forest in the future at the discretion
> of the 9 update team.
>
> Q: For backports for forwardports, will there be a script to translate patch
> files across the consolidation boundary?
> A: That work is planned, but not yet done; see JDK-8165623: Create patch
> translator to update paths pre/post consolidation.
>
> Q: It's the 21st century and I develop using an IDE. That is still going to
> work, right?
> A: The prototype to date does include updating the various IDE support
> files, but bug JDK-8167142 has been filed to track that work.
>