From tom.deneau at amd.com Thu Dec 6 08:36:17 2012 From: tom.deneau at amd.com (Deneau, Tom) Date: Thu, 6 Dec 2012 16:36:17 +0000 Subject: Setting up the Mercurial Repositories for Sumatra Message-ID: How do we get the initial Mercurial repositories created for the Sumatra project? We believe a clone of the http://hg.openjdk.java.net/lambda/lambda forest will be a good starting point. In addition I have a question about branching. Assuming the Sumatra project is going to have a main trunk and also some experimental branches that may or may not get merged into the main trunk, is there a page that explains which of the many branching models supported by Mercurial would be used? >From looking at other projects it looks like separate repositories are made for each branch? If we went that way, what is the process when someone wants to create a new branch? -- Tom Deneau, AMD From tom.deneau at amd.com Thu Dec 6 08:44:16 2012 From: tom.deneau at amd.com (Deneau, Tom) Date: Thu, 6 Dec 2012 16:44:16 +0000 Subject: Issue Tracking Message-ID: Another startup question for Sumatra... What if anything do OpenJDK projects typically use for issue tracking? -- Tom Deneau From brian.goetz at oracle.com Thu Dec 6 08:55:42 2012 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 06 Dec 2012 11:55:42 -0500 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: References: Message-ID: <50C0CE0E.4010902@oracle.com> I think you'd rather just clone from jdk8. Most of the lambda language bits are already there, and the rest will be there soon. At some point long before Sumatra is done, lambda will go away. On 12/6/2012 11:36 AM, Deneau, Tom wrote: > How do we get the initial Mercurial repositories created for the Sumatra project? > > We believe a clone of the http://hg.openjdk.java.net/lambda/lambda forest will be a good starting point. > > In addition I have a question about branching. Assuming the Sumatra project is going to have a main trunk and also some experimental branches that may or may not get merged into the main trunk, is there a page that explains which of the many branching models supported by Mercurial would be used? > >>From looking at other projects it looks like separate repositories are made for each branch? If we went that way, what is the process when someone wants to create a new branch? > > -- Tom Deneau, AMD > From brian.goetz at oracle.com Thu Dec 6 09:01:14 2012 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 06 Dec 2012 12:01:14 -0500 Subject: Issue Tracking In-Reply-To: References: Message-ID: <50C0CF5A.1080201@oracle.com> The story here isn't perfect. The new OpenJDK bug tracker is currently still closed; opening it up is still unfortunately a ways away. Options include: - Wait for openjdk JIRA - Set up a project on java.net and use their JIRA On 12/6/2012 11:44 AM, Deneau, Tom wrote: > Another startup question for Sumatra... > > What if anything do OpenJDK projects typically use for issue tracking? > > -- Tom Deneau > From john.r.rose at oracle.com Fri Dec 7 16:25:01 2012 From: john.r.rose at oracle.com (John Rose) Date: Fri, 7 Dec 2012 16:25:01 -0800 Subject: Issue Tracking In-Reply-To: References: Message-ID: <17E91DF0-7124-4B55-A977-21E2C5131409@oracle.com> On Dec 6, 2012, at 8:44 AM, Deneau, Tom wrote: > Another startup question for Sumatra... > > What if anything do OpenJDK projects typically use for issue tracking? What Brian said is true. If we want a structured issue data base right now, we have to make a java.net JIRA instance. For example, here's my favorite issue in the Lambda EG JIRA project: http://java.net/jira/browse/JSR_335-11 :-) I think that's possible, but I'd like the 335 EG to test those waters first. Anyway, I don't think we need this until we have a few dozen issues to track. For O(30) issues, flat files or wikis work great. Shall we start with simpler tech? In that vein, I just gave you edit rights to this wiki: https://wikis.oracle.com/display/HotSpotInternals/Sumatra To me, it seems convenient to use HotSpotInternals, at least at first, as an option for hosting shared design work that will radically affect hotspot. If we outgrow that box, we can obtain another wiki (or JIRA, if that makes sense). Comments? ? John P.S. Proposal for HG repo coming up next. From john.r.rose at oracle.com Fri Dec 7 16:54:04 2012 From: john.r.rose at oracle.com (John Rose) Date: Fri, 7 Dec 2012 16:54:04 -0800 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: References: Message-ID: On Dec 6, 2012, at 8:36 AM, Deneau, Tom wrote: > How do we get the initial Mercurial repositories created for the Sumatra project? > > We believe a clone of the http://hg.openjdk.java.net/lambda/lambda forest will be a good starting point. Again, what Brian said. (Thanks, Brian.) John Coomes and I have been talking about this for a while, and we have a proposal we'd like to float. It is based on the following observations: A. We need a stable prototype which can be built using standard procedures. B. We need a "sandbox" or "workbench" for maybe-throwaway experiments in modifying the JVM and JDK (not to interfere with A). C. We need a part of the "workbench" to share flat files, small proofs of concept, and other ad hoc resources (not to interfere with A or B). D. We do not need direct mergability into jdk8 (same as lambda repos, which do not merge up directly, and does not need to pass jcheck). E. The workbench version of things needs to be malleable by simultaneous independent workers. F. However, workbench experiments should be shareable, when the workers want to collaborate. G. Successful, maturing experiments should be shareable with a wider audience, by moving changesets to the stable prototoype (A). H. Eventually (as with Project Lambda) the prototype can be moved (after further development) into the main jdk source base (JDK 9 or whichever). I. Unlike big projects like HotSpot, Sumatra will not grow very large, and so does not need a high level of codification and formality. J. We will (per OpenJDK) distinguish the roles of author, committer, and reviewer. In concrete terms, we think this comes to something like the following repositories: A. http://hg.openjdk.java.net/sumatra/sumatra/{hotspot,jdk,langtools,...} (cloned from jdk8, buildable & testable, occasionally refreshed and rebased) B. http://hg.openjdk.java.net/sumatra/sumatra-dev/{hotspot,jdk,langtools} (bundle of independent branches, occasionally rebased on A) C. http://hg.openjdk.java.net/sumatra/sumatra-dev/scratch (initially empty workbench shelf, for sharing artifacts other than JDK source changes) John Coomes is looking into configuring these repos with suitable "jcheck" options to relax some of the OpenJDK rules on changeset structure, and to allow B. to have multiple simultaneous branches. > In addition I have a question about branching. Assuming the Sumatra project is going to have a main trunk and also some experimental branches that may or may not get merged into the main trunk, is there a page that explains which of the many branching models supported by Mercurial would be used? > >> From looking at other projects it looks like separate repositories are made for each branch? If we went that way, what is the process when someone wants to create a new branch? Mechanically, any committer can create a new branch, and the contents of the branch are up to the committer. I would ask that branch names would be briefly reviewed, by Reviewers, before creation, at least at first, so that we build a habit of naming experiments with some consistency. (I'm not thinking of anything heavyweight; see the mlvm repository for some examples.) Before we jump in, though, I have one big worry, and it's the same as yours, Tom: Which practice of branching will work for us? I have enjoyed using the simplest possible version: Flat patch files, handled with MQ (hg qpush etc.) and manually rebased. This gives maximum flexibility, but may be too unfriendly for us. If there is a HG branch model that works better than (versioned) flat patch files, let's use it. If we actually go with flat patch files, then the mlvm provides a starting point for a model. http://hg.openjdk.java.net/mlvm/mlvm/hotspot One more caveat: Whatever branch model we use, it needs to interoperate with the OpenJDK servers, which are an old rev. of HG. Best wishes, ? John From tom.deneau at amd.com Mon Dec 10 08:09:31 2012 From: tom.deneau at amd.com (Deneau, Tom) Date: Mon, 10 Dec 2012 16:09:31 +0000 Subject: Issue Tracking In-Reply-To: <17E91DF0-7124-4B55-A977-21E2C5131409@oracle.com> References: <17E91DF0-7124-4B55-A977-21E2C5131409@oracle.com> Message-ID: John -- We are fine with using the wiki for now... -- Tom From: John Rose [mailto:john.r.rose at oracle.com] Sent: Friday, December 07, 2012 6:25 PM To: Deneau, Tom Cc: sumatra-dev at openjdk.java.net Subject: Re: Issue Tracking On Dec 6, 2012, at 8:44 AM, Deneau, Tom wrote: Another startup question for Sumatra... What if anything do OpenJDK projects typically use for issue tracking? What Brian said is true. If we want a structured issue data base right now, we have to make a java.net JIRA instance. For example, here's my favorite issue in the Lambda EG JIRA project: http://java.net/jira/browse/JSR_335-11 :-) I think that's possible, but I'd like the 335 EG to test those waters first. Anyway, I don't think we need this until we have a few dozen issues to track. For O(30) issues, flat files or wikis work great. Shall we start with simpler tech? In that vein, I just gave you edit rights to this wiki: https://wikis.oracle.com/display/HotSpotInternals/Sumatra To me, it seems convenient to use HotSpotInternals, at least at first, as an option for hosting shared design work that will radically affect hotspot. If we outgrow that box, we can obtain another wiki (or JIRA, if that makes sense). Comments? - John P.S. Proposal for HG repo coming up next. From Gary.Frost at amd.com Mon Dec 10 08:29:16 2012 From: Gary.Frost at amd.com (Frost, Gary) Date: Mon, 10 Dec 2012 16:29:16 +0000 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: References: Message-ID: John I agree with the sumatra, sumatra-dev and sumatra-dev/scratch approach to this. We can then work on pushing our initial Lambda/Aparapi based initial 'tracer bullet' project as a sumatra-dev project . Gary -----Original Message----- From: sumatra-dev-bounces at openjdk.java.net [mailto:sumatra-dev-bounces at openjdk.java.net] On Behalf Of John Rose Sent: Friday, December 07, 2012 6:54 PM To: Deneau, Tom Cc: sumatra-dev at openjdk.java.net Subject: Re: Setting up the Mercurial Repositories for Sumatra On Dec 6, 2012, at 8:36 AM, Deneau, Tom wrote: > How do we get the initial Mercurial repositories created for the Sumatra project? > > We believe a clone of the http://hg.openjdk.java.net/lambda/lambda forest will be a good starting point. Again, what Brian said. (Thanks, Brian.) John Coomes and I have been talking about this for a while, and we have a proposal we'd like to float. It is based on the following observations: A. We need a stable prototype which can be built using standard procedures. B. We need a "sandbox" or "workbench" for maybe-throwaway experiments in modifying the JVM and JDK (not to interfere with A). C. We need a part of the "workbench" to share flat files, small proofs of concept, and other ad hoc resources (not to interfere with A or B). D. We do not need direct mergability into jdk8 (same as lambda repos, which do not merge up directly, and does not need to pass jcheck). E. The workbench version of things needs to be malleable by simultaneous independent workers. F. However, workbench experiments should be shareable, when the workers want to collaborate. G. Successful, maturing experiments should be shareable with a wider audience, by moving changesets to the stable prototoype (A). H. Eventually (as with Project Lambda) the prototype can be moved (after further development) into the main jdk source base (JDK 9 or whichever). I. Unlike big projects like HotSpot, Sumatra will not grow very large, and so does not need a high level of codification and formality. J. We will (per OpenJDK) distinguish the roles of author, committer, and reviewer. In concrete terms, we think this comes to something like the following repositories: A. http://hg.openjdk.java.net/sumatra/sumatra/{hotspot,jdk,langtools,...} (cloned from jdk8, buildable & testable, occasionally refreshed and rebased) B. http://hg.openjdk.java.net/sumatra/sumatra-dev/{hotspot,jdk,langtools} (bundle of independent branches, occasionally rebased on A) C. http://hg.openjdk.java.net/sumatra/sumatra-dev/scratch (initially empty workbench shelf, for sharing artifacts other than JDK source changes) John Coomes is looking into configuring these repos with suitable "jcheck" options to relax some of the OpenJDK rules on changeset structure, and to allow B. to have multiple simultaneous branches. > In addition I have a question about branching. Assuming the Sumatra project is going to have a main trunk and also some experimental branches that may or may not get merged into the main trunk, is there a page that explains which of the many branching models supported by Mercurial would be used? > >> From looking at other projects it looks like separate repositories are made for each branch? If we went that way, what is the process when someone wants to create a new branch? Mechanically, any committer can create a new branch, and the contents of the branch are up to the committer. I would ask that branch names would be briefly reviewed, by Reviewers, before creation, at least at first, so that we build a habit of naming experiments with some consistency. (I'm not thinking of anything heavyweight; see the mlvm repository for some examples.) Before we jump in, though, I have one big worry, and it's the same as yours, Tom: Which practice of branching will work for us? I have enjoyed using the simplest possible version: Flat patch files, handled with MQ (hg qpush etc.) and manually rebased. This gives maximum flexibility, but may be too unfriendly for us. If there is a HG branch model that works better than (versioned) flat patch files, let's use it. If we actually go with flat patch files, then the mlvm provides a starting point for a model. http://hg.openjdk.java.net/mlvm/mlvm/hotspot One more caveat: Whatever branch model we use, it needs to interoperate with the OpenJDK servers, which are an old rev. of HG. Best wishes, - John From tom.deneau at amd.com Mon Dec 10 11:36:57 2012 From: tom.deneau at amd.com (Deneau, Tom) Date: Mon, 10 Dec 2012 19:36:57 +0000 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: References: Message-ID: John -- The A,B,C repository proposal looks fine. For the branching, I am not familiar with the MQ flat patch file procedure (yet another Mercurial branching option!) but if it works with the mlvm project, it will probably work for us. Is there documentation that explains how it is used on mlvm? -- Tom From: John Rose [mailto:john.r.rose at oracle.com] Sent: Friday, December 07, 2012 6:54 PM To: Deneau, Tom Cc: sumatra-dev at openjdk.java.net Subject: Re: Setting up the Mercurial Repositories for Sumatra On Dec 6, 2012, at 8:36 AM, Deneau, Tom wrote: How do we get the initial Mercurial repositories created for the Sumatra project? We believe a clone of the http://hg.openjdk.java.net/lambda/lambda forest will be a good starting point. Again, what Brian said. (Thanks, Brian.) John Coomes and I have been talking about this for a while, and we have a proposal we'd like to float. It is based on the following observations: A. We need a stable prototype which can be built using standard procedures. B. We need a "sandbox" or "workbench" for maybe-throwaway experiments in modifying the JVM and JDK (not to interfere with A). C. We need a part of the "workbench" to share flat files, small proofs of concept, and other ad hoc resources (not to interfere with A or B). D. We do not need direct mergability into jdk8 (same as lambda repos, which do not merge up directly, and does not need to pass jcheck). E. The workbench version of things needs to be malleable by simultaneous independent workers. F. However, workbench experiments should be shareable, when the workers want to collaborate. G. Successful, maturing experiments should be shareable with a wider audience, by moving changesets to the stable prototoype (A). H. Eventually (as with Project Lambda) the prototype can be moved (after further development) into the main jdk source base (JDK 9 or whichever). I. Unlike big projects like HotSpot, Sumatra will not grow very large, and so does not need a high level of codification and formality. J. We will (per OpenJDK) distinguish the roles of author, committer, and reviewer. In concrete terms, we think this comes to something like the following repositories: A. http://hg.openjdk.java.net/sumatra/sumatra/{hotspot,jdk,langtools,...} (cloned from jdk8, buildable & testable, occasionally refreshed and rebased) B. http://hg.openjdk.java.net/sumatra/sumatra-dev/{hotspot,jdk,langtools} (bundle of independent branches, occasionally rebased on A) C. http://hg.openjdk.java.net/sumatra/sumatra-dev/scratch (initially empty workbench shelf, for sharing artifacts other than JDK source changes) John Coomes is looking into configuring these repos with suitable "jcheck" options to relax some of the OpenJDK rules on changeset structure, and to allow B. to have multiple simultaneous branches. In addition I have a question about branching. Assuming the Sumatra project is going to have a main trunk and also some experimental branches that may or may not get merged into the main trunk, is there a page that explains which of the many branching models supported by Mercurial would be used? >From looking at other projects it looks like separate repositories are made for each branch? If we went that way, what is the process when someone wants to create a new branch? Mechanically, any committer can create a new branch, and the contents of the branch are up to the committer. I would ask that branch names would be briefly reviewed, by Reviewers, before creation, at least at first, so that we build a habit of naming experiments with some consistency. (I'm not thinking of anything heavyweight; see the mlvm repository for some examples.) Before we jump in, though, I have one big worry, and it's the same as yours, Tom: Which practice of branching will work for us? I have enjoyed using the simplest possible version: Flat patch files, handled with MQ (hg qpush etc.) and manually rebased. This gives maximum flexibility, but may be too unfriendly for us. If there is a HG branch model that works better than (versioned) flat patch files, let's use it. If we actually go with flat patch files, then the mlvm provides a starting point for a model. http://hg.openjdk.java.net/mlvm/mlvm/hotspot One more caveat: Whatever branch model we use, it needs to interoperate with the OpenJDK servers, which are an old rev. of HG. Best wishes, - John From david.r.chase at oracle.com Mon Dec 10 12:02:18 2012 From: david.r.chase at oracle.com (David Chase) Date: Mon, 10 Dec 2012 15:02:18 -0500 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: References: Message-ID: <6D28A86C-1D61-408F-AA5D-43804CA78EE9@oracle.com> On 2012-12-10, at 2:36 PM, "Deneau, Tom" wrote: > For the branching, I am not familiar with the MQ flat patch file procedure (yet another Mercurial branching option!) but if it works with the mlvm project, it will probably work for us. Is there documentation that explains how it is used on mlvm? There is this: http://hg.openjdk.java.net/mlvm/mlvm/file/1ecfbcb08379/README.txt MQ is also discussed in various semi-official places: http://mercurial.selenic.com/wiki/MqExtension http://hgbook.red-bean.com/read/managing-change-with-mercurial-queues.html http://hgbook.red-bean.com/read/advanced-uses-of-mercurial-queues.html http://hgbook.red-bean.com/read/mercurial-queues-reference.html I am (at Oracle) new to working on JDK, MLVM, and with MQ, though I am pretty happy with Mercurial, and I found this a little intimidating, and in theory it is one of my tasks to clean this up and make it less intimidating. By-the-way, what version of Mercurial are you using? And are you primarily Mac, Linux, or PC people? (If you use a Mac and MacPorts, it turns out that one of the not-well-supported extensions, hg-forest, is part of the package. I have not tested it yet to see if it has really been maintained.) David From John.Coomes at oracle.com Mon Dec 10 17:47:55 2012 From: John.Coomes at oracle.com (John Coomes) Date: Mon, 10 Dec 2012 17:47:55 -0800 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: References: Message-ID: <20678.37067.386362.461868@oracle.com> John Rose (john.r.rose at oracle.com) wrote: > On Dec 6, 2012, at 8:36 AM, Deneau, Tom wrote: > > > How do we get the initial Mercurial repositories created for the Sumatra project? > > > > We believe a clone of the http://hg.openjdk.java.net/lambda/lambda forest will be a good starting point. > > Again, what Brian said. (Thanks, Brian.) > > John Coomes and I have been talking about this for a while, and we have a proposal we'd like to float. It is based on the following observations: > > A. We need a stable prototype which can be built using standard procedures. > B. We need a "sandbox" or "workbench" for maybe-throwaway experiments in modifying the JVM and JDK (not to interfere with A). > C. We need a part of the "workbench" to share flat files, small proofs of concept, and other ad hoc resources (not to interfere with A or B). > D. We do not need direct mergability into jdk8 (same as lambda repos, which do not merge up directly, and does not need to pass jcheck). > E. The workbench version of things needs to be malleable by simultaneous independent workers. > F. However, workbench experiments should be shareable, when the workers want to collaborate. > G. Successful, maturing experiments should be shareable with a wider audience, by moving changesets to the stable prototoype (A). > H. Eventually (as with Project Lambda) the prototype can be moved (after further development) into the main jdk source base (JDK 9 or whichever). > I. Unlike big projects like HotSpot, Sumatra will not grow very large, and so does not need a high level of codification and formality. > J. We will (per OpenJDK) distinguish the roles of author, committer, and reviewer. > > In concrete terms, we think this comes to something like the following repositories: > > A. http://hg.openjdk.java.net/sumatra/sumatra/{hotspot,jdk,langtools,...} (cloned from jdk8, buildable & testable, occasionally refreshed and rebased) > > B. http://hg.openjdk.java.net/sumatra/sumatra-dev/{hotspot,jdk,langtools} (bundle of independent branches, occasionally rebased on A) By "rebase", I assume you are *not* referring to the history rewriting done by the mercurial rebase extension. That sort of history rewriting is very inconvenient in shared repos, as everyone must throw away their old repo, clone the rewritten one and then reapply their work in progress to it. It also requires an admin's help to do that on the servers. > C. http://hg.openjdk.java.net/sumatra/sumatra-dev/scratch (initially empty workbench shelf, for sharing artifacts other than JDK source changes) > > John Coomes is looking into configuring these repos with suitable "jcheck" options to relax some of the OpenJDK rules on changeset structure, and to allow B. to have multiple simultaneous branches. The first (lax checking of changeset comments) is already present in jcheck, and just needs to be enabled for the sumatra repos. The second (allowing named branches) will require a modest change to jcheck and a small change to jprt[1]. > > In addition I have a question about branching. Assuming the Sumatra project is going to have a main trunk and also some experimental branches that may or may not get merged into the main trunk, is there a page that explains which of the many branching models supported by Mercurial would be used? > > > >> From looking at other projects it looks like separate repositories are made for each branch? If we went that way, what is the process when someone wants to create a new branch? > > Mechanically, any committer can create a new branch, and the contents of the branch are up to the committer. I would ask that branch names would be briefly reviewed, by Reviewers, before creation, at least at first, so that we build a habit of naming experiments with some consistency. (I'm not thinking of anything heavyweight; see the mlvm repository for some examples.) > > Before we jump in, though, I have one big worry, and it's the same as yours, Tom: Which practice of branching will work for us? I have enjoyed using the simplest possible version: Flat patch files, handled with MQ (hg qpush etc.) and manually rebased. This gives maximum flexibility, but may be too unfriendly for us. If there is a HG branch model that works better than (versioned) flat patch files, let's use it. Let me list the pros & cons as I see them. Note that I have not worked directly with the mlvm flat patch file model, but have talked to those who have: Flat patch file model (ala mlvm): pros: separation - each patch remains a separate unit for its entire lifetime--until the point it is ready to include in a jdk release repository (e.g., jdk9/jdk9/hotspot), if that's desired. flexibility - can include/exclude various changes as needed, without any commitment to to keeping the result (other than the effort needed to resolve conflicts, if any) cons: conflicts result in patch reject files, which must be fixed up by hand - merge tools are not invoked changeset history is a diff against patch files ("diffs of diffs") requires careful attention to avoid committing or merging changes into the wrong patch many operations are indirect - must clone a stable repo, then apply the patch files before doing the desired operation: build, generate webrev, compare changes, etc. Named branch model: As I envision it, each sumatra feature of decent size would be developed on a separate named branch. During development of the feature, changes from the "stable" branch would regularly be merged into the feature branch (but not the other way around). Once the feature was deemed complete and stable, the feature branch would be closed and the net result applied to the "stable" branch as a single changeset. pros: standard mercurial development model conflicts are resolved using the normal mercurial merge machinery cons: jcheck and jprt[1] would have to be updated to allow named branches (modest changes in both cases). requires careful attention to avoid committing or merging changes on the wrong branch incomplete separation - after the feature is complete and its feature branch is merged into the "stable" branch, the changes required to keep the code in sync with the upstream repos would be scattered across various merge changesets, which would accumlate over time. Thus there would not be a single entity that held the entire change for that feature. the old mercurial version currently in use on our servers would require the use of 'push -f' when creating a new branch or after merging the "stable" branch into a feature branch (newer hg versions don't require this) -John [1] jprt is a continuous integration tool that ensures each change runs a basic set of tests on all platforms before it is pushed to a repo. The main hotspot repos (hsx/hotspot-*) require every change to be run through jprt. Unfortunately, jprt is currently only accessible within Oracle. Sumatra will likely not require jprt, but it is nonetheless a useful tool. > If we actually go with flat patch files, then the mlvm provides a starting point for a model. > http://hg.openjdk.java.net/mlvm/mlvm/hotspot > > One more caveat: Whatever branch model we use, it needs to interoperate with the OpenJDK servers, which are an old rev. of HG. > > Best wishes, > $,1rt(B John From john.r.rose at oracle.com Mon Dec 10 23:34:35 2012 From: john.r.rose at oracle.com (John Rose) Date: Mon, 10 Dec 2012 23:34:35 -0800 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: <20678.37067.386362.461868@oracle.com> References: <20678.37067.386362.461868@oracle.com> Message-ID: <55C6CF35-B200-49A8-A936-1982E5385C68@oracle.com> On Dec 10, 2012, at 5:47 PM, John Coomes wrote: > John Rose (john.r.rose at oracle.com) wrote: >> In concrete terms, we think this comes to something like the following repositories: >> >> A. http://hg.openjdk.java.net/sumatra/sumatra/{hotspot,jdk,langtools,...} (cloned from jdk8, buildable & testable, occasionally refreshed and rebased) >> >> B. http://hg.openjdk.java.net/sumatra/sumatra-dev/{hotspot,jdk,langtools} (bundle of independent branches, occasionally rebased on A) > > By "rebase", I assume you are *not* referring to the history rewriting > done by the mercurial rebase extension. That's correct. I take it as a given that Mercurial history has to be monotonically increasing within any given repo. One advantage of the flat patch file model (as you point out below) is that rebasing a patch set does not require destruction of history. You just generate a new patch (or patches). The effect on repo history is confined to the patch repository, and to the particular patch file that required a rebase. The place where you pay for this separation is if two people try to update the same patch, then you have to merge diffs-of-diffs. That is painful. For this reason, if two people are working on one mlvm patch, we split the patch into parts, and merge it back up later when the concurrent development is finished. > That sort of history > rewriting is very inconvenient in shared repos, as everyone must throw > away their old repo, clone the rewritten one and then reapply their > work in progress to it. It also requires an admin's help to do that > on the servers. Yuck. I suppose part of the problem is that the admin has to create a blacklist of obsolete versions (hashes) which the repo will refuse to accept from out-of-date users. >> C. http://hg.openjdk.java.net/sumatra/sumatra-dev/scratch (initially empty workbench shelf, for sharing artifacts other than JDK source changes) >> >> John Coomes is looking into configuring these repos with suitable "jcheck" options to relax some of the OpenJDK rules on changeset structure, and to allow B. to have multiple simultaneous branches. > > The first (lax checking of changeset comments) is already present in > jcheck, and just needs to be enabled for the sumatra repos. The > second (allowing named branches) will require a modest change to > jcheck and a small change to jprt[1]. My biggest question about branches is, do they work in practice, for the workflows we are intending? (I.e., small scale, provisional & experimental changes, independent workers, controlled mini-integrations.) I don't know anyone who has used them, so I'm slightly doubtful. >> Before we jump in, though, I have one big worry, and it's the same as yours, Tom: Which practice of branching will work for us? I have enjoyed using the simplest possible version: Flat patch files, handled with MQ (hg qpush etc.) and manually rebased. This gives maximum flexibility, but may be too unfriendly for us. If there is a HG branch model that works better than (versioned) flat patch files, let's use it. > > Let me list the pros & cons as I see them. Note that I have not > worked directly with the mlvm flat patch file model, but have talked > to those who have: > > Flat patch file model (ala mlvm): > > pros: > > separation - each patch remains a separate unit for its entire > lifetime--until the point it is ready to include in a jdk > release repository (e.g., jdk9/jdk9/hotspot), if that's > desired. Corollary of separation: No direct interaction between patch history and baseline history. There are rebase events which cause changes to patches, but there are no merge nodes in the baseline repository. Not sure if this is truly a simplification, but it seems so to me. > flexibility - can include/exclude various changes as needed, > without any commitment to to keeping the result (other than > the effort needed to resolve conflicts, if any) Corollary: Patches can be reordered relative to each other, and split or merged. Changes tend to be small, and (again) do not affect baseline repository. > cons: > > conflicts result in patch reject files, which must be fixed up > by hand - merge tools are not invoked There is a claimed solution for this, to the effect that "hg pull --rebase" DTRT on MQ patches: http://stackoverflow.com/questions/11700136/when-doing-qpush-can-i-get-a-merge-tool-instead-of-rej-files I have not tested this. But it seems reasonable. Note that the scary word "rebase" applies to the MQ patches only, which are designed to be rebased, as discussed above. > changeset history is a diff against patch files ("diffs of diffs") Yes. That's the worst. In essence, the changeset history is useful (a) for its comments and other metadata, and (b) as a way to return to an old configuration. The actual diffs-of-diffs ( ? ? ) are hard to read. When I have to evaluate a patch change in mlvm, what I usually do is (a) manually reverse the current version of the patch using "patch -sp1 -R < .hg/patches/foo.patch", (b) materialize the previous version of the patch, (c) run the previous version forward "patch -sp1 -N < .hg/patches/foo.patch.prev, and (d) examine the effect on the working files, using "hg diff". If I had to do it frequently, I'd make it into a script, but I don't. > requires careful attention to avoid committing or merging > changes into the wrong patch (That con applies to both scenarios.) > many operations are indirect - must clone a stable repo, then > apply the patch files before doing the desired operation: > build, generate webrev, compare changes, etc. > > Named branch model: > > As I envision it, each sumatra feature of decent size would be > developed on a separate named branch. During development of the > feature, changes from the "stable" branch would regularly be merged > into the feature branch (but not the other way around). Once the > feature was deemed complete and stable, the feature branch would be > closed and the net result applied to the "stable" branch as a single > changeset. > > pros: > > standard mercurial development model > > conflicts are resolved using the normal mercurial merge > machinery > > cons: > > jcheck and jprt[1] would have to be updated to allow named > branches (modest changes in both cases). > > requires careful attention to avoid committing or merging > changes on the wrong branch > > incomplete separation - after the feature is complete and its > feature branch is merged into the "stable" branch, the changes > required to keep the code in sync with the upstream repos > would be scattered across various merge changesets, which > would accumlate over time. Thus there would not be a single > entity that held the entire change for that feature. Hmm... This problem is related to the one of "cherry-picking" or back-porting, where you want to take some changesets from one repo and move them sideways to another, probably consolidating them into a single changeset. The problem is greatly complicated by the fact that the list of required changesets from the original repo is not easy to compute, if the main changeset has complex dependencies on other changesets, which themselves may contain changes that the back-porter doesn't want. (In practice, I find it difficult to predict how these details of workflow will shake out, which is why I would prefer to rely on somebody's experience. That's why I keep asking about branches.) > the old mercurial version currently in use on our servers > would require the use of 'push -f' when creating a new branch > or after merging the "stable" branch into a feature branch > (newer hg versions don't require this) Thanks, John, for the detailed analysis. It's really interesting. ? John From tom.deneau at amd.com Tue Dec 11 08:14:25 2012 From: tom.deneau at amd.com (Deneau, Tom) Date: Tue, 11 Dec 2012 16:14:25 +0000 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: <55C6CF35-B200-49A8-A936-1982E5385C68@oracle.com> References: <20678.37067.386362.461868@oracle.com> <55C6CF35-B200-49A8-A936-1982E5385C68@oracle.com> Message-ID: Named Branch Experience... At AMD, we have used name branches successfully in our own Hotspot repository clones but we haven't had to face some of the more complicated scenarios outlined below. We generally * start work on a named branch * merge the branch with default whenever we deem necessary * generate the webrev vs. default * keep the named branch open for changes based on webrev reviews until the webrev is actually accepted, we never actually merge the branch into default in our repo, that happens in the hotspot main repo. * close the branch either * after it's been merged in the hotspot main repo * or after the experiment is deemed one that will not go forward. Mostly the branches stand alone and we haven't had the case where changes have to be merged from multiple named branches. -- Tom -----Original Message----- From: John Rose [mailto:john.r.rose at oracle.com] Sent: Tuesday, December 11, 2012 1:35 AM To: John Coomes Cc: Deneau, Tom; sumatra-dev at openjdk.java.net Subject: Re: Setting up the Mercurial Repositories for Sumatra On Dec 10, 2012, at 5:47 PM, John Coomes wrote: > John Rose (john.r.rose at oracle.com) wrote: >> In concrete terms, we think this comes to something like the following repositories: >> >> A. http://hg.openjdk.java.net/sumatra/sumatra/{hotspot,jdk,langtools,...} (cloned from jdk8, buildable & testable, occasionally refreshed and rebased) >> >> B. http://hg.openjdk.java.net/sumatra/sumatra-dev/{hotspot,jdk,langtools} (bundle of independent branches, occasionally rebased on A) > > By "rebase", I assume you are *not* referring to the history rewriting > done by the mercurial rebase extension. That's correct. I take it as a given that Mercurial history has to be monotonically increasing within any given repo. One advantage of the flat patch file model (as you point out below) is that rebasing a patch set does not require destruction of history. You just generate a new patch (or patches). The effect on repo history is confined to the patch repository, and to the particular patch file that required a rebase. The place where you pay for this separation is if two people try to update the same patch, then you have to merge diffs-of-diffs. That is painful. For this reason, if two people are working on one mlvm patch, we split the patch into parts, and merge it back up later when the concurrent development is finished. > That sort of history > rewriting is very inconvenient in shared repos, as everyone must throw > away their old repo, clone the rewritten one and then reapply their > work in progress to it. It also requires an admin's help to do that > on the servers. Yuck. I suppose part of the problem is that the admin has to create a blacklist of obsolete versions (hashes) which the repo will refuse to accept from out-of-date users. >> C. http://hg.openjdk.java.net/sumatra/sumatra-dev/scratch (initially empty workbench shelf, for sharing artifacts other than JDK source changes) >> >> John Coomes is looking into configuring these repos with suitable "jcheck" options to relax some of the OpenJDK rules on changeset structure, and to allow B. to have multiple simultaneous branches. > > The first (lax checking of changeset comments) is already present in > jcheck, and just needs to be enabled for the sumatra repos. The > second (allowing named branches) will require a modest change to > jcheck and a small change to jprt[1]. My biggest question about branches is, do they work in practice, for the workflows we are intending? (I.e., small scale, provisional & experimental changes, independent workers, controlled mini-integrations.) I don't know anyone who has used them, so I'm slightly doubtful. >> Before we jump in, though, I have one big worry, and it's the same as yours, Tom: Which practice of branching will work for us? I have enjoyed using the simplest possible version: Flat patch files, handled with MQ (hg qpush etc.) and manually rebased. This gives maximum flexibility, but may be too unfriendly for us. If there is a HG branch model that works better than (versioned) flat patch files, let's use it. > > Let me list the pros & cons as I see them. Note that I have not > worked directly with the mlvm flat patch file model, but have talked > to those who have: > > Flat patch file model (ala mlvm): > > pros: > > separation - each patch remains a separate unit for its entire > lifetime--until the point it is ready to include in a jdk > release repository (e.g., jdk9/jdk9/hotspot), if that's > desired. Corollary of separation: No direct interaction between patch history and baseline history. There are rebase events which cause changes to patches, but there are no merge nodes in the baseline repository. Not sure if this is truly a simplification, but it seems so to me. > flexibility - can include/exclude various changes as needed, > without any commitment to to keeping the result (other than > the effort needed to resolve conflicts, if any) Corollary: Patches can be reordered relative to each other, and split or merged. Changes tend to be small, and (again) do not affect baseline repository. > cons: > > conflicts result in patch reject files, which must be fixed up > by hand - merge tools are not invoked There is a claimed solution for this, to the effect that "hg pull --rebase" DTRT on MQ patches: http://stackoverflow.com/questions/11700136/when-doing-qpush-can-i-get-a-merge-tool-instead-of-rej-files I have not tested this. But it seems reasonable. Note that the scary word "rebase" applies to the MQ patches only, which are designed to be rebased, as discussed above. > changeset history is a diff against patch files ("diffs of diffs") Yes. That's the worst. In essence, the changeset history is useful (a) for its comments and other metadata, and (b) as a way to return to an old configuration. The actual diffs-of-diffs ( ? ? ) are hard to read. When I have to evaluate a patch change in mlvm, what I usually do is (a) manually reverse the current version of the patch using "patch -sp1 -R < .hg/patches/foo.patch", (b) materialize the previous version of the patch, (c) run the previous version forward "patch -sp1 -N < .hg/patches/foo.patch.prev, and (d) examine the effect on the working files, using "hg diff". If I had to do it frequently, I'd make it into a script, but I don't. > requires careful attention to avoid committing or merging > changes into the wrong patch (That con applies to both scenarios.) > many operations are indirect - must clone a stable repo, then > apply the patch files before doing the desired operation: > build, generate webrev, compare changes, etc. > > Named branch model: > > As I envision it, each sumatra feature of decent size would be > developed on a separate named branch. During development of the > feature, changes from the "stable" branch would regularly be merged > into the feature branch (but not the other way around). Once the > feature was deemed complete and stable, the feature branch would be > closed and the net result applied to the "stable" branch as a single > changeset. > > pros: > > standard mercurial development model > > conflicts are resolved using the normal mercurial merge > machinery > > cons: > > jcheck and jprt[1] would have to be updated to allow named > branches (modest changes in both cases). > > requires careful attention to avoid committing or merging > changes on the wrong branch > > incomplete separation - after the feature is complete and its > feature branch is merged into the "stable" branch, the changes > required to keep the code in sync with the upstream repos > would be scattered across various merge changesets, which > would accumlate over time. Thus there would not be a single > entity that held the entire change for that feature. Hmm... This problem is related to the one of "cherry-picking" or back-porting, where you want to take some changesets from one repo and move them sideways to another, probably consolidating them into a single changeset. The problem is greatly complicated by the fact that the list of required changesets from the original repo is not easy to compute, if the main changeset has complex dependencies on other changesets, which themselves may contain changes that the back-porter doesn't want. (In practice, I find it difficult to predict how these details of workflow will shake out, which is why I would prefer to rely on somebody's experience. That's why I keep asking about branches.) > the old mercurial version currently in use on our servers > would require the use of 'push -f' when creating a new branch > or after merging the "stable" branch into a feature branch > (newer hg versions don't require this) Thanks, John, for the detailed analysis. It's really interesting. ? John From brian.goetz at oracle.com Tue Dec 11 08:17:23 2012 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 11 Dec 2012 11:17:23 -0500 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: References: <20678.37067.386362.461868@oracle.com> <55C6CF35-B200-49A8-A936-1982E5385C68@oracle.com> Message-ID: <50C75C93.8030802@oracle.com> We had some issues using named branches in lambda, which we think was due to the old version of Mercurial that OpenJDK uses. YMMV. On 12/11/2012 11:14 AM, Deneau, Tom wrote: > Named Branch Experience... > > At AMD, we have used name branches successfully in our own Hotspot repository clones but we haven't had to face some of the more complicated scenarios outlined below. We generally > * start work on a named branch > * merge the branch with default whenever we deem necessary > * generate the webrev vs. default > * keep the named branch open for changes based on webrev reviews until the webrev is actually accepted, we never actually merge the branch into default in our repo, that happens in the hotspot main repo. > * close the branch either > * after it's been merged in the hotspot main repo > * or after the experiment is deemed one that will not go forward. > > Mostly the branches stand alone and we haven't had the case where changes have to be merged from multiple named branches. > > -- Tom > > > > -----Original Message----- > From: John Rose [mailto:john.r.rose at oracle.com] > Sent: Tuesday, December 11, 2012 1:35 AM > To: John Coomes > Cc: Deneau, Tom; sumatra-dev at openjdk.java.net > Subject: Re: Setting up the Mercurial Repositories for Sumatra > > On Dec 10, 2012, at 5:47 PM, John Coomes wrote: > >> John Rose (john.r.rose at oracle.com) wrote: >>> In concrete terms, we think this comes to something like the following repositories: >>> >>> A. http://hg.openjdk.java.net/sumatra/sumatra/{hotspot,jdk,langtools,...} (cloned from jdk8, buildable & testable, occasionally refreshed and rebased) >>> >>> B. http://hg.openjdk.java.net/sumatra/sumatra-dev/{hotspot,jdk,langtools} (bundle of independent branches, occasionally rebased on A) >> >> By "rebase", I assume you are *not* referring to the history rewriting >> done by the mercurial rebase extension. > > That's correct. I take it as a given that Mercurial history has to be monotonically increasing within any given repo. > > One advantage of the flat patch file model (as you point out below) is that rebasing a patch set does not require destruction of history. You just generate a new patch (or patches). The effect on repo history is confined to the patch repository, and to the particular patch file that required a rebase. > > The place where you pay for this separation is if two people try to update the same patch, then you have to merge diffs-of-diffs. That is painful. For this reason, if two people are working on one mlvm patch, we split the patch into parts, and merge it back up later when the concurrent development is finished. > >> That sort of history >> rewriting is very inconvenient in shared repos, as everyone must throw >> away their old repo, clone the rewritten one and then reapply their >> work in progress to it. It also requires an admin's help to do that >> on the servers. > > Yuck. I suppose part of the problem is that the admin has to create a blacklist of obsolete versions (hashes) which the repo will refuse to accept from out-of-date users. > >>> C. http://hg.openjdk.java.net/sumatra/sumatra-dev/scratch (initially empty workbench shelf, for sharing artifacts other than JDK source changes) >>> >>> John Coomes is looking into configuring these repos with suitable "jcheck" options to relax some of the OpenJDK rules on changeset structure, and to allow B. to have multiple simultaneous branches. >> >> The first (lax checking of changeset comments) is already present in >> jcheck, and just needs to be enabled for the sumatra repos. The >> second (allowing named branches) will require a modest change to >> jcheck and a small change to jprt[1]. > > My biggest question about branches is, do they work in practice, for the workflows we are intending? (I.e., small scale, provisional & experimental changes, independent workers, controlled mini-integrations.) I don't know anyone who has used them, so I'm slightly doubtful. > >>> Before we jump in, though, I have one big worry, and it's the same as yours, Tom: Which practice of branching will work for us? I have enjoyed using the simplest possible version: Flat patch files, handled with MQ (hg qpush etc.) and manually rebased. This gives maximum flexibility, but may be too unfriendly for us. If there is a HG branch model that works better than (versioned) flat patch files, let's use it. >> >> Let me list the pros & cons as I see them. Note that I have not >> worked directly with the mlvm flat patch file model, but have talked >> to those who have: >> >> Flat patch file model (ala mlvm): >> >> pros: >> >> separation - each patch remains a separate unit for its entire >> lifetime--until the point it is ready to include in a jdk >> release repository (e.g., jdk9/jdk9/hotspot), if that's >> desired. > > Corollary of separation: No direct interaction between patch history and baseline history. There are rebase events which cause changes to patches, but there are no merge nodes in the baseline repository. Not sure if this is truly a simplification, but it seems so to me. > >> flexibility - can include/exclude various changes as needed, >> without any commitment to to keeping the result (other than >> the effort needed to resolve conflicts, if any) > > Corollary: Patches can be reordered relative to each other, and split or merged. Changes tend to be small, and (again) do not affect baseline repository. > >> cons: >> >> conflicts result in patch reject files, which must be fixed up >> by hand - merge tools are not invoked > > There is a claimed solution for this, to the effect that "hg pull --rebase" DTRT on MQ patches: > http://stackoverflow.com/questions/11700136/when-doing-qpush-can-i-get-a-merge-tool-instead-of-rej-files > > I have not tested this. But it seems reasonable. Note that the scary word "rebase" applies to the MQ patches only, which are designed to be rebased, as discussed above. > >> changeset history is a diff against patch files ("diffs of diffs") > > Yes. That's the worst. In essence, the changeset history is useful (a) for its comments and other metadata, and (b) as a way to return to an old configuration. The actual diffs-of-diffs ( ? ? ) are hard to read. > > When I have to evaluate a patch change in mlvm, what I usually do is (a) manually reverse the current version of the patch using "patch -sp1 -R < .hg/patches/foo.patch", (b) materialize the previous version of the patch, (c) run the previous version forward "patch -sp1 -N < .hg/patches/foo.patch.prev, and (d) examine the effect on the working files, using "hg diff". If I had to do it frequently, I'd make it into a script, but I don't. > >> requires careful attention to avoid committing or merging >> changes into the wrong patch > > (That con applies to both scenarios.) > >> many operations are indirect - must clone a stable repo, then >> apply the patch files before doing the desired operation: >> build, generate webrev, compare changes, etc. >> >> Named branch model: >> >> As I envision it, each sumatra feature of decent size would be >> developed on a separate named branch. During development of the >> feature, changes from the "stable" branch would regularly be merged >> into the feature branch (but not the other way around). Once the >> feature was deemed complete and stable, the feature branch would be >> closed and the net result applied to the "stable" branch as a single >> changeset. >> >> pros: >> >> standard mercurial development model >> >> conflicts are resolved using the normal mercurial merge >> machinery >> >> cons: >> >> jcheck and jprt[1] would have to be updated to allow named >> branches (modest changes in both cases). >> >> requires careful attention to avoid committing or merging >> changes on the wrong branch >> >> incomplete separation - after the feature is complete and its >> feature branch is merged into the "stable" branch, the changes >> required to keep the code in sync with the upstream repos >> would be scattered across various merge changesets, which >> would accumlate over time. Thus there would not be a single >> entity that held the entire change for that feature. > > Hmm... This problem is related to the one of "cherry-picking" or back-porting, where you want to take some changesets from one repo and move them sideways to another, probably consolidating them into a single changeset. The problem is greatly complicated by the fact that the list of required changesets from the original repo is not easy to compute, if the main changeset has complex dependencies on other changesets, which themselves may contain changes that the back-porter doesn't want. > > (In practice, I find it difficult to predict how these details of workflow will shake out, which is why I would prefer to rely on somebody's experience. That's why I keep asking about branches.) > >> the old mercurial version currently in use on our servers >> would require the use of 'push -f' when creating a new branch >> or after merging the "stable" branch into a feature branch >> (newer hg versions don't require this) > > Thanks, John, for the detailed analysis. It's really interesting. > > ? John > From john.r.rose at oracle.com Tue Dec 11 13:55:33 2012 From: john.r.rose at oracle.com (John Rose) Date: Tue, 11 Dec 2012 13:55:33 -0800 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: References: <20678.37067.386362.461868@oracle.com> <55C6CF35-B200-49A8-A936-1982E5385C68@oracle.com> Message-ID: On Dec 11, 2012, at 8:14 AM, Deneau, Tom wrote: > Named Branch Experience... > > At AMD, we have used name branches successfully in our own Hotspot repository clones but we haven't had to face some of the more complicated scenarios outlined below. We generally > * start work on a named branch > * merge the branch with default whenever we deem necessary > * generate the webrev vs. default > * keep the named branch open for changes based on webrev reviews until the webrev is actually accepted, we never actually merge the branch into default in our repo, that happens in the hotspot main repo. > * close the branch either > * after it's been merged in the hotspot main repo > * or after the experiment is deemed one that will not go forward. > > Mostly the branches stand alone and we haven't had the case where changes have to be merged from multiple named branches. That's encouraging. A couple of questions (clarifications, really, since I think the answers are all 'yes'): 1. Is this the case: Your branches can be pushed to your group repo and shared easily among people working on the branch? 2. Is this also the case: People working independently on other branches can share the same group repo, without cross-branch interference? 3. When merging up to a new default, the nice merge tools kick in, right? 4. Independent branches can merge up independently, right? About patches vs. branches: If branch A needs to be merged into another branch B, we could convert branch A to an explicit artifact A.patch at that point, and then import A.patch, either using MQ or permanently into B. The artifact B.patch would provide a clear and separable export point. Put another way, I don't think branches exclude patches, and vice versa. I'm thinking that we can set up the machinery to allow both workflows, and then gravitate towards whatever works. About patches: I like patches, because I feel more control over single text files, backed by single threads of history. But the control requires special patch-oriented tools, notably Emacs diff-mode, so it's not for everybody. And dealing with *.rej files is a pain, though Emacs helps a lot, and I'm used to it. Unsolved question: Is there an MQ-centric way of getting the nice HG merge tools to assist with merge-ups to default? (In the case of patches, as noted earlier, such merge-ups are really rebasing, but without the strange effects on repository history.) If there is, then the *.rej files can disappear from the discussion. Proposal: Let's enable the branches, and agree to admit patches also, keeping an eye on workability, with the reasonable hope of converging on best practices as we go. ? John From tom.deneau at amd.com Tue Dec 11 14:42:15 2012 From: tom.deneau at amd.com (Deneau, Tom) Date: Tue, 11 Dec 2012 22:42:15 +0000 Subject: Setting up the Mercurial Repositories for Sumatra In-Reply-To: References: <20678.37067.386362.461868@oracle.com> <55C6CF35-B200-49A8-A936-1982E5385C68@oracle.com> Message-ID: John -- The answers to #1 thru #4 are yes. We haven't often had need to merge Branch A with Branch B, but I would think that would work. -- Tom -----Original Message----- From: John Rose [mailto:john.r.rose at oracle.com] Sent: Tuesday, December 11, 2012 3:56 PM To: Deneau, Tom Cc: John Coomes; sumatra-dev at openjdk.java.net Subject: Re: Setting up the Mercurial Repositories for Sumatra On Dec 11, 2012, at 8:14 AM, Deneau, Tom wrote: > Named Branch Experience... > > At AMD, we have used name branches successfully in our own Hotspot repository clones but we haven't had to face some of the more complicated scenarios outlined below. We generally > * start work on a named branch > * merge the branch with default whenever we deem necessary > * generate the webrev vs. default > * keep the named branch open for changes based on webrev reviews until the webrev is actually accepted, we never actually merge the branch into default in our repo, that happens in the hotspot main repo. > * close the branch either > * after it's been merged in the hotspot main repo > * or after the experiment is deemed one that will not go forward. > > Mostly the branches stand alone and we haven't had the case where changes have to be merged from multiple named branches. That's encouraging. A couple of questions (clarifications, really, since I think the answers are all 'yes'): 1. Is this the case: Your branches can be pushed to your group repo and shared easily among people working on the branch? 2. Is this also the case: People working independently on other branches can share the same group repo, without cross-branch interference? 3. When merging up to a new default, the nice merge tools kick in, right? 4. Independent branches can merge up independently, right? About patches vs. branches: If branch A needs to be merged into another branch B, we could convert branch A to an explicit artifact A.patch at that point, and then import A.patch, either using MQ or permanently into B. The artifact B.patch would provide a clear and separable export point. Put another way, I don't think branches exclude patches, and vice versa. I'm thinking that we can set up the machinery to allow both workflows, and then gravitate towards whatever works. About patches: I like patches, because I feel more control over single text files, backed by single threads of history. But the control requires special patch-oriented tools, notably Emacs diff-mode, so it's not for everybody. And dealing with *.rej files is a pain, though Emacs helps a lot, and I'm used to it. Unsolved question: Is there an MQ-centric way of getting the nice HG merge tools to assist with merge-ups to default? (In the case of patches, as noted earlier, such merge-ups are really rebasing, but without the strange effects on repository history.) If there is, then the *.rej files can disappear from the discussion. Proposal: Let's enable the branches, and agree to admit patches also, keeping an eye on workability, with the reasonable hope of converging on best practices as we go. - John From Vasanth.Venkatachalam at amd.com Wed Dec 12 08:51:06 2012 From: Vasanth.Venkatachalam at amd.com (Venkatachalam, Vasanth) Date: Wed, 12 Dec 2012 16:51:06 +0000 Subject: changes to Hotspot to support multiple targets Message-ID: <5DD1503F815BD14889DC81D28643E3A7329CC1B2@sausexdag06.amd.com> Hotspot supports generating code for a single ISA (e.g., x86) at a time. However, to have the flexibility of executing code on either the CPU or the GPU, the compilers and code generators for both targets would need to coexist and communicate with each other. Do people have ideas on how this change can be implemented in Hotspot? Vasanth -- Vasanth Venkatachalam AMD Runtimes (512)602-6177 From Vasanth.Venkatachalam at amd.com Thu Dec 13 15:46:02 2012 From: Vasanth.Venkatachalam at amd.com (Venkatachalam, Vasanth) Date: Thu, 13 Dec 2012 23:46:02 +0000 Subject: adding a new ISA Message-ID: <5DD1503F815BD14889DC81D28643E3A7329CC64E@sausexdag06.amd.com> Modifying the JVM to support a new ISA (such as the GPU) would require extensive changes. I was wondering what people's thoughts were on the following: 1) What guidelines do people use for deciding which components of the JVM to start modifying or writing code for? It seems like a good guideline would be to start with the changes that are the most straightforward, such as register definitions or the assembler. Do people have other ideas? 2) How do people incrementally test components that depend upon the rest of the software which isn't in place yet? For example, if you update the register allocator and nothing else, how would you test this change without having the rest of the JVM in place? Vasanth -- Vasanth Venkatachalam AMD Runtimes (512)602-6177 From pcpratts at chirrup.org Fri Dec 14 14:22:34 2012 From: pcpratts at chirrup.org (Phil Pratt-Szeliga) Date: Fri, 14 Dec 2012 17:22:34 -0500 Subject: adding a new ISA In-Reply-To: <5DD1503F815BD14889DC81D28643E3A7329CC64E@sausexdag06.amd.com> References: <5DD1503F815BD14889DC81D28643E3A7329CC64E@sausexdag06.amd.com> Message-ID: Hello, I am looking into moving to using LLVM for Rootbeer. It has the beginnings of a ptx backend [1-2]. The course I teach next semester (Software Studio at Syracuse) will focus on making the LLVM ptx backend everything that Rootbeer needs to pass its tests. Phil Pratt-Szeliga Syracuse University http://chirrup.org/ [1] http://llvm.org/devmtg/2011-11/Holewinski_PTXBackend.pdf [2] http://llvm.org/docs/CodeGenerator.html#the-ptx-backend On Thu, Dec 13, 2012 at 6:46 PM, Venkatachalam, Vasanth wrote: > Modifying the JVM to support a new ISA (such as the GPU) would require extensive changes. > I was wondering what people's thoughts were on the following: > > > 1) What guidelines do people use for deciding which components of the JVM to start modifying or writing code for? It seems like a good guideline would be to start with the changes that are the most straightforward, such as register definitions or the assembler. Do people have other ideas? > > 2) How do people incrementally test components that depend upon the rest of the software which isn't in place yet? For example, if you update the register allocator and nothing else, how would you test this change without having the rest of the JVM in place? > > Vasanth > > -- > Vasanth Venkatachalam > AMD Runtimes > (512)602-6177 > From ivan.p.krylov at gmail.com Sun Dec 16 12:16:20 2012 From: ivan.p.krylov at gmail.com (Ivan Krylov) Date: Mon, 17 Dec 2012 00:16:20 +0400 Subject: changes to Hotspot to support multiple targets In-Reply-To: <5DD1503F815BD14889DC81D28643E3A7329CC1B2@sausexdag06.amd.com> References: <5DD1503F815BD14889DC81D28643E3A7329CC1B2@sausexdag06.amd.com> Message-ID: <50CE2C14.6020107@gmail.com> This is indeed a non-typical target for the Hotspot. AFAIK, Hotspot does not support heterogeneous targets of any kind. I'd suggest to implement as a first step support for the interpreter-only that would force compilation for the GPU-suitable code (whever is used to mark a given method or branch "GPU-suitable"). While for x86 the transition from interpreter to compiled code happens with OSR (on-stack replacement) in case of sumatra it would be a call to the SC to start execution. For some simple compute-intensive examples this could be a good proof of concept because the host code gets called just once and will be interpreted anyway and the compute-intensive code gets compilet for GPS and executed for GPU. At the later stage a new target would need be set up (call it HSA). Thanks, Ivan On 12-Dec-12 20:51, Venkatachalam, Vasanth wrote: > Hotspot supports generating code for a single ISA (e.g., x86) at a time. However, to have the flexibility of executing code on either the CPU or the GPU, the compilers and code generators for both targets would need to coexist and communicate with each other. > > Do people have ideas on how this change can be implemented in Hotspot? > > Vasanth > > > -- > Vasanth Venkatachalam > AMD Runtimes > (512)602-6177 > From Ryan.LaMothe at pnnl.gov Mon Dec 17 11:28:27 2012 From: Ryan.LaMothe at pnnl.gov (LaMothe, Ryan R) Date: Mon, 17 Dec 2012 11:28:27 -0800 Subject: Was: Syncup call Now: Requirements? In-Reply-To: Message-ID: Thanks Gary. Sorry for the long delay in following-up on this email thread. I found the conference call to be enlightening and I have been really interested in continuing that conversation. Surprisingly, there has been a distinct lack of discussion on this mailing list related to points discussed in that conference call, but I think there are some important things that really need to be discussed. I warn that this is a long email, if this information is best in a wiki, then please let me know. First, I apologize for the mis-statements on JSR-308 in my previous email. For all of the important points brought up in the email, that one sentence drove significant on-list and off-list discussion. I meant to address "statements on annotations", which is a powerful approach many non-Java languages have taken to address parallel compute problems. Second, there seems to be a lot of angst (both direct and indirect) regarding such items as "uncertainty whether we can make get this working in the JVM", "let's not discuss language features or things can quickly derail", etc. but I very, very strongly urge this team to pause for a few moments and answer some very basic questions: 1. Who are our customer(s)? 2. What problem(s) are we trying to solve? 3. How is our solution better than the competition? I have looked at https://wikis.oracle.com/display/HotSpotInternals/Sumatra but did not see the answers to these questions. I understand this may feel non-Agile and we all just want to write code, but we really need to answer the three questions above, very specifically, before we do that. I believe Project Sumatra to be incredibly important to the future of Java and the polyglot JVM, but I'd hate to look up from writing code months or years from now and realize we created a solution to a problem that didn't exist or worse, the wrong solution to the wrong problems. I will take a crack at what I think the answers may be for Project Sumatra, but I really want to hear what others think as well: 1. I believe our initial customers will be scientists, engineers and quants. I do not believe that our customers will be "average joe" programmers until parallel computing becomes a mainstream concept and/or is taught in undergraduate college courses. Long-term we need to address everyone. 2. I believe this has to be answered both "top down" and "bottom up" (with overlap). Right now, it seems like people are focused on "how do I expose the hardware" or "how do I execute code on the hardware" but does that really solve the problem or answer the question? For example, we already have existing libraries that can do both of those things and if needed, non-Java programming languages we can call via JNI. I believe the most fundamental problem we're trying to address is "general purpose parallel programming and execution on heterogeneous hardware in Java". For the top-down problem(s), I believe they are the following: * Provide a natural and intuitive interface for parallel programming, addressing both task parallel and data parallel paradigms * I will address these points in more detail below * Provide as many built-in libraries for parallel computation as possible (BLAS, etc.) * Make the underlying hardware execution as transparent as possible to the programmer * If we have to expose "local" or "private" memory access, group sizes, wavefronts, etc. to the user have we failed? * Should we provide both a course-grained and a fine-grained interface? * Provide non-trivial parallel computation support on heterogeneous hardware * NBody is interesting, but most real-world problems are incredibly more complex and these are the problems we should be targeting * Can we document specific computational problems we'd like to solve using Project Sumatra? * Support synchronous and asynchronous execution models * As environments become increasingly multi-threaded, multi-process and distributed, can we provide a robust asynchronous execution model? * If not, can we even provide a basic asynchronous model? * Support unsigned types including unsigned doubles * This is particularly important for scientists, engineers and quants * The lack of unsigned types, especially unsigned doubles, causes huge headaches for folks trying to perform highly accurate computation on GPUs via Java (not to be confused with precision) For the bottom-up problem(s), I believe they are the following (many of these are on the Aparapi mailing list and issue tracking system): * Provide nontrivial parallel computation on heterogeneous hardware * Support synchronous and asynchronous execution models * In order to achieve acceptable throughput on hardware such as GPUs and MICs, concepts such as "overlapping compute", "task queueing" and "out-of-order execution" become paramount * Can we support a "job" or "task" queueing system? * Can we support assigning work to more than one hardware device in a physical system from either the same queue or multiple queues? * Can we support low-level technologies that may help increase throughput between multiple CPUs and GPUs such as "dynamic parallelism"? * Support system resource management and scheduling * When systems begin to have multiple heterogenous compute units, resource management becomes essential * Can we support a global resource manager for all hardware in a system? For example multiple CPUs/cores, multiple GPUs, automated data transfers and data management, dependencies, etc. * Can we make this resource manager extensible? * Support vectorized types such as float4, float8, etc. * This is particularly important for scientists, engineers * How can we effectively utilize parallel vector hardware like GPUs without vectorized types? * Support multiple entry-points during computations * This is required by many computational algorithms that need results from previous executions to either stay on the GPU so they can be used by the next algorithm or copied to a second GPU for execution while new execution starts on the first GPU * In OpenCL terminology this can be either "multiple entry points per kernel" or "multiple kernels" sharing the same data and results * Support computation and graphical display on the same hardware as the data * This is required to visualize "big data" * The current implementation is only partially functional using OpenCL + OpenGL * Can we do better or help fix the current implementations? * Is this even possible in Java? * This is a real world problem that many people are currently working to address 3. How much existing literature and "state-of-the-art" review have we done to compare our new wheel to existing wheels? * How have existing Java-like languages addressed parallel computation? * Microsoft .NET has provided an excellent solution to both task and data Parallel Programming (http://msdn.microsoft.com/en-us/library/dd460693.aspx) * In my personal opinion, this should be a high-value target for Project Sumatra * How have existing non-Java languages addressed parallel computation? * In many non-Java languages, compiler directives on statements are welcome. State-of-the-art solutions such as OpenACC (http://www.openacc-standard.org) embrace a number of advanced options for heterogeneous and parallel computation * Microsoft C++ AMP provides a powerful parallel computation framework (http://msdn.microsoft.com/en-us/library/vstudio/hh265137.aspx) * Microsoft C++ AMP also supports "big data" visualization (http://msdn.microsoft.com/en-us/library/vstudio/hh913015.aspx) * What about other frameworks for data and task parallel compute on hybrid computing platforms such as StarPU (http://runtime.bordeaux.inria.fr/StarPU) * StarPU also transparently supports distributed computation via MPI * I was surprised during the conference call when I asked if we were planning to work with the Java Fork/Join folks for this effort and the answer was No * Java is late to the parallel programming scene and the Fork/Join framework is Java's first official attempt at parallel programming (that I know of) * The Fork/Join framework is Java's closest answer to the .NET solution above, but provides only a subset of .NET's solution * What additional/external efforts currently exist for Java parallel programming? * A number of excellent examples spring to mind: * Aparapi (http://code.google.com/p/aparapi/) * JSR-166 and Extra166y Parallel Arrays (http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/extra166y/) * Parallel.For/Parallel.ForEach (http://www.cs.gmu.edu/~rcarver/cs475/Parallel.java) * JPPF for parallel distributed grid computing (http://www.jppf.org/) * Part of the brilliance of Aparapi is the ability to have "automated fallback" for code which fails to execute on parallel hardware like GPUs * Should we investigate implementing Extra166y or Parallel.For exactly as written except using Aparapi, default to GPU execution and fallback to existing Java code? * This should be fairly easy to create a working proof-of-concept There is much more to discuss and document, I look forward to the discussion. __________________________________________________ Ryan LaMothe On 11/14/12 6:18 AM, "Frost, Gary" wrote: >Ryan, > >There is some great information here. Sumatra can clearly benefit from >your feedback from using Aparapi 'in the trenches'. >You are right to point out that Aparapi essentially blocks until the GPU >compute has returned, this was necessary due to limitations of pinning >memory from the GC. Once we have finer control of memory (via Sumatra >under the cover in the JVM) these limitations (and many others) should be >surmountable, and we should be able to provide a much richer execution >model. However, I think (once again) the opportunities for some simpler >and more optimal models will become obvious as we dive into the >Collections+Lambda APIs. >