code tools proposal - trees extension

[John Coomes]

Name:  trees

Summary:  a mercurial extension for managing loosely-coupled nested
repositories

Proposed by:  John Coomes, hsx Project Lead

Rationale:

The OpenJDK source code is (thankfully) split into multiple
repositories--currently 8.  Many OpenJDK developers work with only a
subset of the repositories at a time, but often more than one.  The
forest extension has been the primary method for working with multiple
repositories to date; it allows all or some of the repositories to be
treated as a loosely-coupled unit.  However, it is relatively
inefficient, requiring a traversal of the filesystem to discover nested
repositories on each operation, it is unsupported, and is very closely
tied to mercurial internals.

The trees extension provides the flexibility of working with subsets of
the repositories while avoiding the frequent filesystem scans done by
the forest extension.  It provides access to more commands that operate
on multiple repositories, the ability to select subsets of the repos on
the command line, and includes a test suite with good coverage based on
mercurial's run-tests.py (it has been tested with mercurial versions
from 1.6 - 2.5.4).  It also changes the dependence on mercurial
internals to rely primarily on the commands api, which has historically
been more stable.  It has been used within Oracle for more than a year;
the built-in help appears below.

One possible alternative is the "subrepo" feature included in recent
versions of mercurial, which allows tracking revisions across
repositories.  However, it is designed around a very tightly-coupled
model, which does not provide support for working with a partial set of
repos, and which makes combining repositories from different servers
difficult (something done frequently within Oracle, and presumably
elsewhere).

trees built-in help:

trees extension - manage loosely-coupled nested repositories

A 'tree' is simply a repository that may contain other repositories (or other
trees) nested within its working directory.  This extension provides commands
that operate on an entire tree, or on selected trees within it.

Each tree stores a list of the repositories contained within it in a non-
versioned file, .hg/trees.  The file lists the path to each contained tree,
relative to the root, one per line.  The file is created when a tree is
cloned, and can be modified using the tconfig command.  It is not updated when
pushing or pulling changesets (tpush, tpull).

The extension is usable on the client even if the hg server does not have the
trees extension enabled; it simply requires a bit more typing when cloning
repos.  Each repo in the tree maintains its own path information in .hg/hgrc,
so that repositories from different locations can be combined into a single
tree.

The following example creates a tree called 'myproj' that includes four nested
repositories ('src', 'docs', 'images' and 'styles') with the last two coming
from a different server.  If the hg servers have the trees extension enabled:

  $ hg tclone http://abc/proj myproj
  $ hg tclone --skiproot http://xyz/pub myproj

If the hg servers do not have the trees extension enabled, then simply append
the desired contained repos (subtrees) to the command line:

  $ hg tclone http://abc/proj myproj src docs
  $ hg tclone --skiproot http://xyz/pub myproj images styles

The above is shorthand for five clone operations (three from the first
command, and two more from the second):

  $ hg clone http://abc/proj       myproj
  $ hg clone http://abc/proj/src   myproj/src
  $ hg clone http://abc/proj/docs  myproj/docs
  $ hg clone http://xyz/pub/images myproj/images
  $ hg clone http://xyz/pub/styles myproj/styles

It also writes the tree configuration to myproj/.hg/trees.  Note that the same
tree can also be created with a single tclone:

  $ hg tclone http://abc/proj myproj src docs http://xyz/pub images styles

More examples:

Show the working directory status for each repo in the tree:

  $ hg tstatus

Pull upstream changes into each repo in the tree and update the working dir:

  $ hg tpull -u

Push local changes from each repo in the tree:

  $ hg tpush

List the tree configuration recursively:

  $ hg tlist
  /path/to/myproj
  /path/to/myproj/src
  /path/to/myproj/docs
  /path/to/myproj/images
  /path/to/myproj/styles

You can create abbreviations for tree lists by adding a [trees] section to
your hgrc file, e.g.:

  [trees]
  jdk-lt = jdk langtools
  ojdk-all = corba hotspot jaxp jaxws jdk-lt

which could be used like this:

    $ hg tclone http://hg.openjdk.java.net/jdk7/jdk7 myjdk7 ojdk-all

to create the myjdk7 tree which contains corba, hotspot, jaxp, jaxws, jdk and
langtools repos.

Show the working directory status, but only for the root repo plus the jdk and
langtools repos:

  $ hg tstatus --subtrees jdk-lt

list of commands:

 tclone        copy one or more existing repositories to create a tree
 tcommand      run a command in each repo in the tree
 tcommit       commit all files
 tconfig       list or change the subtrees configuration
 tdebugkeys    list the tree configuration using mercurial's pushkey
               mechanism
 tdefpath      examine and manipulate default path settings for a tree
 tdiff         diff repository (or selected files)
 theads        show current repository heads or show branch heads
 tincoming     show new changesets found in source
 tlist         list the repo and configured subtrees, recursively
 tlog          show revision history of entire repository or files
 tmerge        merge working directory with another revision
 toutgoing     show changesets not found in the destination
 tparents      (no help text available)
 tpaths        show aliases for remote repositories
 tpull         pull changes from the specified source
 tpush         push changes to the specified destination
 tstatus       show changed files in the working directory
 tsummary      summarize working directory state
 ttag          add one or more tags for the current or given revision
 ttip          show the tip revision
 tupdate       update working directory (or switch revisions)
 tversion      show version information

use "hg -v help trees" to show builtin aliases and global options