From mark.reinhold at oracle.com Mon Apr 1 15:38:42 2013 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 01 Apr 2013 18:38:42 -0400 Subject: CFV: New Code Tools Committer: Aleksey Shipilev In-Reply-To: <42c04cd6-dcb9-4cad-a9ab-ceabd915dc36@default> References: <42c04cd6-dcb9-4cad-a9ab-ceabd915dc36@default> Message-ID: <20130401183842.126857@eggemoggin.niobe.net> Vote: yes - Mark From John.Coomes at oracle.com Tue Apr 9 17:20:02 2013 From: John.Coomes at oracle.com (John Coomes) Date: Tue, 9 Apr 2013 17:20:02 -0700 Subject: code tools proposal - trees extension Message-ID: <20836.45106.272640.956245@oracle.com> Name: trees Summary: a mercurial extension for managing loosely-coupled nested repositories Proposed by: John Coomes, hsx Project Lead Rationale: The OpenJDK source code is (thankfully) split into multiple repositories--currently 8. Many OpenJDK developers work with only a subset of the repositories at a time, but often more than one. The forest extension has been the primary method for working with multiple repositories to date; it allows all or some of the repositories to be treated as a loosely-coupled unit. However, it is relatively inefficient, requiring a traversal of the filesystem to discover nested repositories on each operation, it is unsupported, and is very closely tied to mercurial internals. The trees extension provides the flexibility of working with subsets of the repositories while avoiding the frequent filesystem scans done by the forest extension. It provides access to more commands that operate on multiple repositories, the ability to select subsets of the repos on the command line, and includes a test suite with good coverage based on mercurial's run-tests.py (it has been tested with mercurial versions from 1.6 - 2.5.4). It also changes the dependence on mercurial internals to rely primarily on the commands api, which has historically been more stable. It has been used within Oracle for more than a year; the built-in help appears below. One possible alternative is the "subrepo" feature included in recent versions of mercurial, which allows tracking revisions across repositories. However, it is designed around a very tightly-coupled model, which does not provide support for working with a partial set of repos, and which makes combining repositories from different servers difficult (something done frequently within Oracle, and presumably elsewhere). trees built-in help: trees extension - manage loosely-coupled nested repositories A 'tree' is simply a repository that may contain other repositories (or other trees) nested within its working directory. This extension provides commands that operate on an entire tree, or on selected trees within it. Each tree stores a list of the repositories contained within it in a non- versioned file, .hg/trees. The file lists the path to each contained tree, relative to the root, one per line. The file is created when a tree is cloned, and can be modified using the tconfig command. It is not updated when pushing or pulling changesets (tpush, tpull). The extension is usable on the client even if the hg server does not have the trees extension enabled; it simply requires a bit more typing when cloning repos. Each repo in the tree maintains its own path information in .hg/hgrc, so that repositories from different locations can be combined into a single tree. The following example creates a tree called 'myproj' that includes four nested repositories ('src', 'docs', 'images' and 'styles') with the last two coming from a different server. If the hg servers have the trees extension enabled: $ hg tclone http://abc/proj myproj $ hg tclone --skiproot http://xyz/pub myproj If the hg servers do not have the trees extension enabled, then simply append the desired contained repos (subtrees) to the command line: $ hg tclone http://abc/proj myproj src docs $ hg tclone --skiproot http://xyz/pub myproj images styles The above is shorthand for five clone operations (three from the first command, and two more from the second): $ hg clone http://abc/proj myproj $ hg clone http://abc/proj/src myproj/src $ hg clone http://abc/proj/docs myproj/docs $ hg clone http://xyz/pub/images myproj/images $ hg clone http://xyz/pub/styles myproj/styles It also writes the tree configuration to myproj/.hg/trees. Note that the same tree can also be created with a single tclone: $ hg tclone http://abc/proj myproj src docs http://xyz/pub images styles More examples: Show the working directory status for each repo in the tree: $ hg tstatus Pull upstream changes into each repo in the tree and update the working dir: $ hg tpull -u Push local changes from each repo in the tree: $ hg tpush List the tree configuration recursively: $ hg tlist /path/to/myproj /path/to/myproj/src /path/to/myproj/docs /path/to/myproj/images /path/to/myproj/styles You can create abbreviations for tree lists by adding a [trees] section to your hgrc file, e.g.: [trees] jdk-lt = jdk langtools ojdk-all = corba hotspot jaxp jaxws jdk-lt which could be used like this: $ hg tclone http://hg.openjdk.java.net/jdk7/jdk7 myjdk7 ojdk-all to create the myjdk7 tree which contains corba, hotspot, jaxp, jaxws, jdk and langtools repos. Show the working directory status, but only for the root repo plus the jdk and langtools repos: $ hg tstatus --subtrees jdk-lt list of commands: tclone copy one or more existing repositories to create a tree tcommand run a command in each repo in the tree tcommit commit all files tconfig list or change the subtrees configuration tdebugkeys list the tree configuration using mercurial's pushkey mechanism tdefpath examine and manipulate default path settings for a tree tdiff diff repository (or selected files) theads show current repository heads or show branch heads tincoming show new changesets found in source tlist list the repo and configured subtrees, recursively tlog show revision history of entire repository or files tmerge merge working directory with another revision toutgoing show changesets not found in the destination tparents (no help text available) tpaths show aliases for remote repositories tpull pull changes from the specified source tpush push changes to the specified destination tstatus show changed files in the working directory tsummary summarize working directory state ttag add one or more tags for the current or given revision ttip show the tip revision tupdate update working directory (or switch revisions) tversion show version information use "hg -v help trees" to show builtin aliases and global options From iris.clark at oracle.com Wed Apr 10 11:34:21 2013 From: iris.clark at oracle.com (Iris Clark) Date: Wed, 10 Apr 2013 11:34:21 -0700 (PDT) Subject: Result: New Code Tools Committer: Aleksey Shipilev In-Reply-To: <42c04cd6-dcb9-4cad-a9ab-ceabd915dc36@default> References: <42c04cd6-dcb9-4cad-a9ab-ceabd915dc36@default> Message-ID: Voting for Aleksey Shipilev [1] is now closed. Yes: 3 Veto: 0 Abstain: 0 According to the Bylaws definition of Lazy Consensus, this is sufficient to approve the nomination. Congratulations and welcome Aleksey! Iris Clark [1]: http://mail.openjdk.java.net/pipermail/code-tools-dev/2013-March/000005.html From jonathan.gibbons at oracle.com Wed Apr 10 11:46:45 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 10 Apr 2013 11:46:45 -0700 Subject: code tools proposal - trees extension In-Reply-To: <20836.45106.272640.956245@oracle.com> References: <20836.45106.272640.956245@oracle.com> Message-ID: <5165B395.70501@oracle.com> John, This sounds like a great tool. If anyone has any comments, please post them by the end of Wednesday, 17 April. -- Jon On 04/09/2013 05:20 PM, John Coomes wrote: > Name: trees > > Summary: a mercurial extension for managing loosely-coupled nested > repositories > > Proposed by: John Coomes, hsx Project Lead > > Rationale: > > The OpenJDK source code is (thankfully) split into multiple > repositories--currently 8. Many OpenJDK developers work with only a > subset of the repositories at a time, but often more than one. The > forest extension has been the primary method for working with multiple > repositories to date; it allows all or some of the repositories to be > treated as a loosely-coupled unit. However, it is relatively > inefficient, requiring a traversal of the filesystem to discover nested > repositories on each operation, it is unsupported, and is very closely > tied to mercurial internals. > > The trees extension provides the flexibility of working with subsets of > the repositories while avoiding the frequent filesystem scans done by > the forest extension. It provides access to more commands that operate > on multiple repositories, the ability to select subsets of the repos on > the command line, and includes a test suite with good coverage based on > mercurial's run-tests.py (it has been tested with mercurial versions > from 1.6 - 2.5.4). It also changes the dependence on mercurial > internals to rely primarily on the commands api, which has historically > been more stable. It has been used within Oracle for more than a year; > the built-in help appears below. > > One possible alternative is the "subrepo" feature included in recent > versions of mercurial, which allows tracking revisions across > repositories. However, it is designed around a very tightly-coupled > model, which does not provide support for working with a partial set of > repos, and which makes combining repositories from different servers > difficult (something done frequently within Oracle, and presumably > elsewhere). > > trees built-in help: > > trees extension - manage loosely-coupled nested repositories > > A 'tree' is simply a repository that may contain other repositories (or other > trees) nested within its working directory. This extension provides commands > that operate on an entire tree, or on selected trees within it. > > Each tree stores a list of the repositories contained within it in a non- > versioned file, .hg/trees. The file lists the path to each contained tree, > relative to the root, one per line. The file is created when a tree is > cloned, and can be modified using the tconfig command. It is not updated when > pushing or pulling changesets (tpush, tpull). > > The extension is usable on the client even if the hg server does not have the > trees extension enabled; it simply requires a bit more typing when cloning > repos. Each repo in the tree maintains its own path information in .hg/hgrc, > so that repositories from different locations can be combined into a single > tree. > > The following example creates a tree called 'myproj' that includes four nested > repositories ('src', 'docs', 'images' and 'styles') with the last two coming > from a different server. If the hg servers have the trees extension enabled: > > $ hg tclone http://abc/proj myproj > $ hg tclone --skiproot http://xyz/pub myproj > > If the hg servers do not have the trees extension enabled, then simply append > the desired contained repos (subtrees) to the command line: > > $ hg tclone http://abc/proj myproj src docs > $ hg tclone --skiproot http://xyz/pub myproj images styles > > The above is shorthand for five clone operations (three from the first > command, and two more from the second): > > $ hg clone http://abc/proj myproj > $ hg clone http://abc/proj/src myproj/src > $ hg clone http://abc/proj/docs myproj/docs > $ hg clone http://xyz/pub/images myproj/images > $ hg clone http://xyz/pub/styles myproj/styles > > It also writes the tree configuration to myproj/.hg/trees. Note that the same > tree can also be created with a single tclone: > > $ hg tclone http://abc/proj myproj src docs http://xyz/pub images styles > > More examples: > > Show the working directory status for each repo in the tree: > > $ hg tstatus > > Pull upstream changes into each repo in the tree and update the working dir: > > $ hg tpull -u > > Push local changes from each repo in the tree: > > $ hg tpush > > List the tree configuration recursively: > > $ hg tlist > /path/to/myproj > /path/to/myproj/src > /path/to/myproj/docs > /path/to/myproj/images > /path/to/myproj/styles > > You can create abbreviations for tree lists by adding a [trees] section to > your hgrc file, e.g.: > > [trees] > jdk-lt = jdk langtools > ojdk-all = corba hotspot jaxp jaxws jdk-lt > > which could be used like this: > > $ hg tclone http://hg.openjdk.java.net/jdk7/jdk7 myjdk7 ojdk-all > > to create the myjdk7 tree which contains corba, hotspot, jaxp, jaxws, jdk and > langtools repos. > > Show the working directory status, but only for the root repo plus the jdk and > langtools repos: > > $ hg tstatus --subtrees jdk-lt > > list of commands: > > tclone copy one or more existing repositories to create a tree > tcommand run a command in each repo in the tree > tcommit commit all files > tconfig list or change the subtrees configuration > tdebugkeys list the tree configuration using mercurial's pushkey > mechanism > tdefpath examine and manipulate default path settings for a tree > tdiff diff repository (or selected files) > theads show current repository heads or show branch heads > tincoming show new changesets found in source > tlist list the repo and configured subtrees, recursively > tlog show revision history of entire repository or files > tmerge merge working directory with another revision > toutgoing show changesets not found in the destination > tparents (no help text available) > tpaths show aliases for remote repositories > tpull pull changes from the specified source > tpush push changes to the specified destination > tstatus show changed files in the working directory > tsummary summarize working directory state > ttag add one or more tags for the current or given revision > ttip show the tip revision > tupdate update working directory (or switch revisions) > tversion show version information > > use "hg -v help trees" to show builtin aliases and global options From jonathan.gibbons at oracle.com Wed Apr 10 12:44:30 2013 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 10 Apr 2013 12:44:30 -0700 Subject: code tools proposal - trees extension In-Reply-To: <20836.45106.272640.956245@oracle.com> References: <20836.45106.272640.956245@oracle.com> Message-ID: <5165C11E.7010600@oracle.com> I guess one very general question I would have is, how would this relate to the existing scripts like get_source.sh and hgforest.sh -- Jon On 04/09/2013 05:20 PM, John Coomes wrote: > Name: trees > > Summary: a mercurial extension for managing loosely-coupled nested > repositories > > Proposed by: John Coomes, hsx Project Lead > > Rationale: > > The OpenJDK source code is (thankfully) split into multiple > repositories--currently 8. Many OpenJDK developers work with only a > subset of the repositories at a time, but often more than one. The > forest extension has been the primary method for working with multiple > repositories to date; it allows all or some of the repositories to be > treated as a loosely-coupled unit. However, it is relatively > inefficient, requiring a traversal of the filesystem to discover nested > repositories on each operation, it is unsupported, and is very closely > tied to mercurial internals. > > The trees extension provides the flexibility of working with subsets of > the repositories while avoiding the frequent filesystem scans done by > the forest extension. It provides access to more commands that operate > on multiple repositories, the ability to select subsets of the repos on > the command line, and includes a test suite with good coverage based on > mercurial's run-tests.py (it has been tested with mercurial versions > from 1.6 - 2.5.4). It also changes the dependence on mercurial > internals to rely primarily on the commands api, which has historically > been more stable. It has been used within Oracle for more than a year; > the built-in help appears below. > > One possible alternative is the "subrepo" feature included in recent > versions of mercurial, which allows tracking revisions across > repositories. However, it is designed around a very tightly-coupled > model, which does not provide support for working with a partial set of > repos, and which makes combining repositories from different servers > difficult (something done frequently within Oracle, and presumably > elsewhere). > > trees built-in help: > > trees extension - manage loosely-coupled nested repositories > > A 'tree' is simply a repository that may contain other repositories (or other > trees) nested within its working directory. This extension provides commands > that operate on an entire tree, or on selected trees within it. > > Each tree stores a list of the repositories contained within it in a non- > versioned file, .hg/trees. The file lists the path to each contained tree, > relative to the root, one per line. The file is created when a tree is > cloned, and can be modified using the tconfig command. It is not updated when > pushing or pulling changesets (tpush, tpull). > > The extension is usable on the client even if the hg server does not have the > trees extension enabled; it simply requires a bit more typing when cloning > repos. Each repo in the tree maintains its own path information in .hg/hgrc, > so that repositories from different locations can be combined into a single > tree. > > The following example creates a tree called 'myproj' that includes four nested > repositories ('src', 'docs', 'images' and 'styles') with the last two coming > from a different server. If the hg servers have the trees extension enabled: > > $ hg tclone http://abc/proj myproj > $ hg tclone --skiproot http://xyz/pub myproj > > If the hg servers do not have the trees extension enabled, then simply append > the desired contained repos (subtrees) to the command line: > > $ hg tclone http://abc/proj myproj src docs > $ hg tclone --skiproot http://xyz/pub myproj images styles > > The above is shorthand for five clone operations (three from the first > command, and two more from the second): > > $ hg clone http://abc/proj myproj > $ hg clone http://abc/proj/src myproj/src > $ hg clone http://abc/proj/docs myproj/docs > $ hg clone http://xyz/pub/images myproj/images > $ hg clone http://xyz/pub/styles myproj/styles > > It also writes the tree configuration to myproj/.hg/trees. Note that the same > tree can also be created with a single tclone: > > $ hg tclone http://abc/proj myproj src docs http://xyz/pub images styles > > More examples: > > Show the working directory status for each repo in the tree: > > $ hg tstatus > > Pull upstream changes into each repo in the tree and update the working dir: > > $ hg tpull -u > > Push local changes from each repo in the tree: > > $ hg tpush > > List the tree configuration recursively: > > $ hg tlist > /path/to/myproj > /path/to/myproj/src > /path/to/myproj/docs > /path/to/myproj/images > /path/to/myproj/styles > > You can create abbreviations for tree lists by adding a [trees] section to > your hgrc file, e.g.: > > [trees] > jdk-lt = jdk langtools > ojdk-all = corba hotspot jaxp jaxws jdk-lt > > which could be used like this: > > $ hg tclone http://hg.openjdk.java.net/jdk7/jdk7 myjdk7 ojdk-all > > to create the myjdk7 tree which contains corba, hotspot, jaxp, jaxws, jdk and > langtools repos. > > Show the working directory status, but only for the root repo plus the jdk and > langtools repos: > > $ hg tstatus --subtrees jdk-lt > > list of commands: > > tclone copy one or more existing repositories to create a tree > tcommand run a command in each repo in the tree > tcommit commit all files > tconfig list or change the subtrees configuration > tdebugkeys list the tree configuration using mercurial's pushkey > mechanism > tdefpath examine and manipulate default path settings for a tree > tdiff diff repository (or selected files) > theads show current repository heads or show branch heads > tincoming show new changesets found in source > tlist list the repo and configured subtrees, recursively > tlog show revision history of entire repository or files > tmerge merge working directory with another revision > toutgoing show changesets not found in the destination > tparents (no help text available) > tpaths show aliases for remote repositories > tpull pull changes from the specified source > tpush push changes to the specified destination > tstatus show changed files in the working directory > tsummary summarize working directory state > ttag add one or more tags for the current or given revision > ttip show the tip revision > tupdate update working directory (or switch revisions) > tversion show version information > > use "hg -v help trees" to show builtin aliases and global options