Setting up the Mercurial Repositories for Sumatra

[John Coomes]

John Rose (john.r.rose at oracle.com) wrote:
> On Dec 6, 2012, at 8:36 AM, Deneau, Tom wrote:
> 
> > How do we get the initial Mercurial repositories created for the Sumatra project? 
> > 
> > We believe a clone of the http://hg.openjdk.java.net/lambda/lambda forest will be a good starting point.
> 
> Again, what Brian said.  (Thanks, Brian.)
> 
> John Coomes and I have been talking about this for a while, and we have a proposal we'd like to float.  It is based on the following observations:
> 
> A. We need a stable prototype which can be built using standard procedures.
> B. We need a "sandbox" or "workbench" for maybe-throwaway experiments in modifying the JVM and JDK (not to interfere with A).
> C. We need a part of the "workbench" to share flat files, small proofs of concept, and other ad hoc resources (not to interfere with A or B).
> D. We do not need direct mergability into jdk8 (same as lambda repos, which do not merge up directly, and does not need to pass jcheck).
> E. The workbench version of things needs to be malleable by simultaneous independent workers.
> F. However, workbench experiments should be shareable, when the workers want to collaborate.
> G. Successful, maturing experiments should be shareable with a wider audience, by moving changesets to the stable prototoype (A).
> H. Eventually (as with Project Lambda) the prototype can be moved (after further development) into the main jdk source base (JDK 9 or whichever).
> I. Unlike big projects like HotSpot, Sumatra will not grow very large, and so does not need a high level of codification and formality.
> J. We will (per OpenJDK) distinguish the roles of author, committer, and reviewer.
> 
> In concrete terms, we think this comes to something like the following repositories:
> 
> A. http://hg.openjdk.java.net/sumatra/sumatra/{hotspot,jdk,langtools,...}  (cloned from jdk8, buildable & testable, occasionally refreshed and rebased)
> 
> B. http://hg.openjdk.java.net/sumatra/sumatra-dev/{hotspot,jdk,langtools}  (bundle of independent branches, occasionally rebased on A)

By "rebase", I assume you are *not* referring to the history rewriting
done by the mercurial rebase extension.  That sort of history
rewriting is very inconvenient in shared repos, as everyone must throw
away their old repo, clone the rewritten one and then reapply their
work in progress to it.  It also requires an admin's help to do that
on the servers.

> C. http://hg.openjdk.java.net/sumatra/sumatra-dev/scratch (initially empty workbench shelf, for sharing artifacts other than JDK source changes)
> 
> John Coomes is looking into configuring these repos with suitable "jcheck" options to relax some of the OpenJDK rules on changeset structure, and to allow B. to have multiple simultaneous branches.

The first (lax checking of changeset comments) is already present in
jcheck, and just needs to be enabled for the sumatra repos.  The
second (allowing named branches) will require a modest change to
jcheck and a small change to jprt[1].

> > In addition I have a question about branching.  Assuming the Sumatra project is going to have a main trunk and also some experimental branches that may or may not get merged into the main trunk, is there a page that explains which of the many branching models supported by Mercurial would be used?
> > 
> >> From looking at other projects it looks like separate repositories are made for each branch?  If we went that way, what is the process when someone wants to create a new branch?
> 
> Mechanically, any committer can create a new branch, and the contents of the branch are up to the committer.  I would ask that branch names would be briefly reviewed, by Reviewers, before creation, at least at first, so that we build a habit of naming experiments with some consistency.  (I'm not thinking of anything heavyweight; see the mlvm repository for some examples.)
> 
> Before we jump in, though, I have one big worry, and it's the same as yours, Tom:  Which practice of branching will work for us?  I have enjoyed using the simplest possible version:  Flat patch files, handled with MQ (hg qpush etc.) and manually rebased.  This gives maximum flexibility, but may be too unfriendly for us.  If there is a HG branch model that works better than (versioned) flat patch files, let's use it.

Let me list the pros & cons as I see them.  Note that I have not
worked directly with the mlvm flat patch file model, but have talked
to those who have:

Flat patch file model (ala mlvm):

pros:

	separation - each patch remains a separate unit for its entire
	lifetime--until the point it is ready to include in a jdk
	release repository (e.g., jdk9/jdk9/hotspot), if that's
	desired.

	flexibility - can include/exclude various changes as needed,
	without any commitment to to keeping the result (other than
	the effort needed to resolve conflicts, if any)

cons:

	conflicts result in patch reject files, which must be fixed up
	by hand - merge tools are not invoked

	changeset history is a diff against patch files ("diffs of diffs")

	requires careful attention to avoid committing or merging
	changes into the wrong patch

	many operations are indirect - must clone a stable repo, then
	apply the patch files before doing the desired operation:
	build, generate webrev, compare changes, etc.

Named branch model:

As I envision it, each sumatra feature of decent size would be
developed on a separate named branch.  During development of the
feature, changes from the "stable" branch would regularly be merged
into the feature branch (but not the other way around).  Once the
feature was deemed complete and stable, the feature branch would be
closed and the net result applied to the "stable" branch as a single
changeset.

pros:

	standard mercurial development model

	conflicts are resolved using the normal mercurial merge
	machinery

cons:

	jcheck and jprt[1] would have to be updated to allow named
	branches (modest changes in both cases).

	requires careful attention to avoid committing or merging
	changes on the wrong branch

	incomplete separation - after the feature is complete and its
	feature branch is merged into the "stable" branch, the changes
	required to keep the code in sync with the upstream repos
	would be scattered across various merge changesets, which
	would accumlate over time.  Thus there would not be a single
	entity that held the entire change for that feature.

	the old mercurial version currently in use on our servers
	would require the use of 'push -f' when creating a new branch
	or after merging the "stable" branch into a feature branch
	(newer hg versions don't require this)

-John

[1] jprt is a continuous integration tool that ensures each change
runs a basic set of tests on all platforms before it is pushed to a
repo.  The main hotspot repos (hsx/hotspot-*) require every change to
be run through jprt.  Unfortunately, jprt is currently only accessible
within Oracle.  Sumatra will likely not require jprt, but it is
nonetheless a useful tool.

> If we actually go with flat patch files, then the mlvm provides a starting point for a model.
>   http://hg.openjdk.java.net/mlvm/mlvm/hotspot
> 
> One more caveat:  Whatever branch model we use, it needs to interoperate with the OpenJDK servers, which are an old rev. of HG.
> 
> Best wishes,
> $,1rt(B John