Looking ahead: proposed Hg forest consolidation for JDK 10

Lindenmaier, Goetz goetz.lindenmaier at sap.com
Mon Oct 17 12:22:07 UTC 2016 

Hi,

I'm on a 1.6 TB share available to our team visible on all our
servers. The machine is a 48 processor 64GB linux x86_64 server. 
The servers only have limited local disc space.  Especially with the
new setup I can not have clones on all the machines I need to compile 
and test on.

Best regards,
  Goetz.


> -----Original Message-----
> From: Erik Helin [mailto:erik.helin at oracle.com]
> Sent: Montag, 17. Oktober 2016 13:42
> To: Lindenmaier, Goetz <goetz.lindenmaier at sap.com>
> Cc: Anthony Vanelverdinghe <anthony.vanelverdinghe at gmail.com>; jdk9-
> dev at openjdk.java.net
> Subject: Re: Looking ahead: proposed Hg forest consolidation for JDK 10
> 
> On 2016-10-17, Lindenmaier, Goetz wrote:
> > Hi Anthony and Joe,
> >
> > > such as hg rebase, hg histedit, hg graft, hg strip, hg strip --keep, and hg
> commit --amend" [7].
> > I understand how these commands replace hg queues.
> > I don't understand how they replace having several
> > clones to do something like this:
> >  -  Change 1 is compiling  (in clone 1)
> >  -  Debugging change 2 (in clone 2)
> >  - I get a review and want to immediately edit change 3 (in clone 3), without
> >    invalidating the sources I stepped in the debugging session and without
> >    canceling the builds.
> 
> I would recommend two similar but slighty different workflows for this:
> - local clones (as previously discussed). One local clone per
>   feature/bug/review/debugging-session.
> - shares, using `hg share` and bookmarks. One bookmark in each hg share.
> 
> > If I use a single clone, I can have these three changes in three branches,
> > or in several  sequential changes (like a mercurial queue) I keep reordering
> with
> > histedit. But as there is only one source tree I can only work on one of
> > the changes at a time.
> > Hg share will do this job I assume.
> >
> > I have been looking at the consol-proto:
> 
> The timings seems an order of magnitude off? What kind of machine are
> you using? See inline for my measurements, all done on a 1 year old
> Samsung SSD with an ext4 filesystem and Linux kernel 4.3.3. I'm using hg
> version 3.8.1.
> 
> > hg clone takes 30 minutes.  Before, get_source.sh took 20 mins.
> 
> $ time hg clone http://hg.openjdk.java.net/jdk9/consol-proto
> requesting all changes
> adding changesets
> adding manifests
> adding file changes
> added 41157 changesets with 358201 changes to 148305 files
> updating to branch default
> 53435 files updated, 0 files merged, 0 files removed, 0 files unresolved
> 
> real    17m42.057s
> user    4m37.156s
> sys     0m41.153s
> 
> The time for the remote clone will depend mostly on yours (and
> Oracle's) network. The amount of bits that needs to be transferred are
> almost the same (differs on a few MB IIRC) compared to a forest. The
> difference is that the forest downloads the metadata for multiple
> repositories in parallel.
> 
> I know you find it cumbersome, but my recommendation here would be to
> not clone that often from the remote servers. Since mercurial is a DVCS,
> you already have most of the bits locally on your machine if you've
> already cloned once.
> 
> > Hg share takes 5 minutes. Before, hg clone of hotspot took 3 mins.
> 
> $ hg clone http://hg.openjdk.java.net/jdk9/consol-proto
> $ time hg share --bookmarks consol-proto share
> updating working directory
> 53435 files updated, 0 files merged, 0 files removed, 0 files unresolved
> 
> real    0m7.528s
> user    0m28.442s
> sys     0m9.791s
> 
> I have no idea why it takes 5 (or even 3) minutes on your machine?
> 
> > Hg diff takes 32 secs!!!, before it took 4 secs on hotspot repo.
> 
> $ hg clone http://hg.openjdk.java.net/jdk9/consol-proto
> $ cd consol-proto/src/hotspot/
> $ wget http://cr.openjdk.java.net/~goetz/wr16/8166560-
> basic_s390/hotspot.wr04/hotspot.changeset
> $ patch -p2 hotspot.changeset # skip changes to jdk.hotspot.agent
> $ time hg diff
> M src/hotspot/os/linux/vm/os_linux.cpp
> M src/hotspot/share/tools/hsdis/hsdis.c
> M src/hotspot/share/vm/code/codeCache.cpp
> M src/hotspot/share/vm/interpreter/abstractInterpreter.hpp
> M src/hotspot/share/vm/runtime/globals.hpp
> M src/hotspot/share/vm/runtime/vm_version.cpp
> M src/hotspot/share/vm/utilities/macros.hpp
> ? src/hotspot/hotspot.changeset
> ? src/hotspot/share/vm/interpreter/abstractInterpreter.hpp.orig
> ? src/hotspot/share/vm/runtime/globals.hpp.orig
> 
> real    0m0.787s
> user    0m0.587s
> sys     0m0.200s
> 
> What patch/changes did you apply before running `hg diff`? I have hard
> time getting `hg diff` to take longer than 1 second...
> 
> > The full repo requires 1.9G.
> 
> $ hg clone http://hg.openjdk.java.net/jdk9/consol-proto
> $ du -ms consol-proto
> 1754    consol-proto/
> 
> I don't know why yours is 191 MB larger than mine. Different filesystem
> and or hg version?
> 
> > A 'hg share' repo requires 0.6G
> 
> $ mkdir measurements && cd measurements
> $ hg clone http://hg.openjdk.java.net/jdk9/consol-proto
> $ du -ms .
> 1754    .
> 
> $ hg share console-proto share
> $ du -ms .
> 2415    .
> 
> so 2415 - 1754 = 661 MB for a share, so similar to my measurements.
> 
> Thanks,
> Erik
> 
> > A hotspot repo before required 0.2G.
> >
> > I will be able to live with this using modern, slower functionality ;)
> > But it imposes a considerable overhead in hardware, tool runtime
> > and administration on my side.
> >
> > Best regards,
> >   Goetz.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Can I check out several
> >
> >
> >
> > > -----Original Message-----
> > > From: jdk9-dev [mailto:jdk9-dev-bounces at openjdk.java.net] On Behalf
> Of
> > > Anthony Vanelverdinghe
> > > Sent: Freitag, 14. Oktober 2016 22:11
> > > To: jdk9-dev at openjdk.java.net
> > > Subject: Re: Looking ahead: proposed Hg forest consolidation for JDK 10
> > >
> > > Hi
> > >
> > > While I'm not an OpenJDK committer (yet, hope to become so one fine
> > > day), I believe this is a great initiative. Since several people have
> > > raised concerns related to the increased repository size, I just wanted
> > > to point out that both Facebook and Mozilla work with Mercurial
> > > repositories which dwarf a typical OpenJDK repository. For example, a
> > > clone of mozilla-central [1] is 2.79 GB, whereas a clone of jdk9-dev is
> > > less than 1 GB.
> > >
> > > There are also several Mercurial extensions which may prove to be useful
> > > for people having to work with large repositories and/or adapt their
> > > workflows.
> > > During their work to scale Mercurial [2], Facebook contributed/made
> > > several extensions, such as fsmonitor as mentioned by Joe [3], and the
> > > ones in their BitBucket repository [4], such as remotefilelog [5].
> > > When working with local clones, the share extension may be helpful [6].
> > > Finally, note that mq is "often considered for deprecation", so this may
> > > be an opportunity to adopt "modern tools, such as hg rebase, hg
> > > histedit, hg graft, hg strip, hg strip --keep, and hg commit --amend" [7].
> > >
> > > Kind regards,
> > > Anthony
> > >
> > > [1] https://hg.mozilla.org/mozilla-central/
> > > [2]
> > > https://code.facebook.com/posts/218678814984400/scaling-mercurial-
> at-
> > > facebook/
> > > [3] http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-
> > > October/004990.html
> > > [4] https://bitbucket.org/facebook/hg-experimental
> > > [5]
> > > https://bitbucket.org/facebook/hg-
> > >
> experimental/src/d2c3a2c02eb6c7e5a7331ba0cf15e5bf7c8dc8dc/remotefilel
> > > og/?at=default
> > > [6] https://www.mercurial-scm.org/wiki/ShareExtension
> > > [7] https://www.mercurial-scm.org/wiki/MqExtension
> > >
> > > On 14/10/2016 19:03, Brian Goetz wrote:
> > > >
> > > >> Conversely, I think it is reasonable for engineers making changes to
> > > >> the JDK to be wiling to offer some flexibility in adjusting
> > > >> established worksflows optimized for the split repositories to
> > > >> accommodate the sort of infrastructure changes being proposed here
> > > >> for a consolidated one.
> > > >
> > > > Let me amplify this: OpenJDK developers are not the only stakeholders
> > > > here.   By aligning more with the way the rest of the world develops
> > > > -- all code in one linearized, transactionally updated repo -- it
> > > > increases the feasibility / reduces the cost of tools like 'bisect' to
> > > > determine where a fault was introduced. This reduces SQE costs and
> > > > increases product quality -- something we all have a stake in.  David
> > > > Lloyd has pointed up other tooling-related benefits, such as making it
> > > > easier to maintain a git mirror.
> > > >
> > > > Most of the objections raised so far have been "(I think they will)
> > > > make my life harder."  Fair enough; people should be their own
> > > > advocates.  But let's not forget the significant benefits that accrue
> > > > to *everyone* as a result, and keep those in mind when judging the
> > > > pros and cons.
> > > >
> > > >
> > > >
> >

More information about the jdk9-dev mailing list