Mercurial clonebundles for hg.openjdk.java.net

[Aleksey Shipilev]

Hi,

This might not be under the umbrella of Skara project, and it may be irrelevant if/when we migrate
to Git, but can we repeat what Mozilla does for their monorepo and enable clonebundles [1] on
hg.openjdk.java.net, at least on busy repositories?

It seems to be easy to configure: enable extension on server side, "hg bundle" the current state of
repository, put the bundle somewhere, list the bundles in .hg/clonebundles.manifest on server side,
and everything works magically from there. Clients would choose the bundle they can understand, or
fallback to regular clone. Clients would also pull all the new changesets after the bundle was
created, which means bundles can be refreshed only once in a while.

My dirty experiments with clonebundles are very promising. Even within-Europe clones have improved
quite a bit [2][3], almost down to the clone performance of local seeding [4]. I think that means:
a) clonebundles test bottlenecks on hg client unbundling performance; b) non-clonebundles test
bottlenecks on hgserver CPU. With clonebundles, hgserver CPU utilization drops to almost zero.

The bundle is streamed by the usual HTTP(S) servers, which can be the part of geo-local CDN. Of
course, doing similar thing with Git and Skara's GitHub mirror is much faster [5], quite probably
because Git server+client is faster at dealing with the repo.

Bottom-line: Enabling clonebundles would simplify our development lives before Skara arrives, and it
would certainly be usable on repositories which Skara migration would not touch (e.g. all
jdk-updates/*?). It should get CPU costs for hg.openjdk.java.net down too.

Thanks,
-Aleksey

[1] https://www.mercurial-scm.org/wiki/ClonebundlesExtension

[2] ------- remote hgserver, clonebundles disabled

$ time hg clone --config ui.clonebundles=false http://shipilev.net:8080/ jdk-jdk
requesting all changes
adding changesets
adding manifests
adding file changes
added 51849 changesets with 448820 changes to 180327 files


new changesets fd16c54261b3:54afe70c50b6
updating to branch default
66512 files updated, 0 files merged, 0 files removed, 0 files unresolved



real	7m18.963s
user	2m27.424s
sys	0m16.004s

[3] ------- remote hgserver, clonebundles enabled, remote seed

$ time hg clone http://shipilev.net:8080/ jdk-jdk
applying clone bundle from https://builds.shipilev.net/scratch/jdk-jdk.zstd-max.hg
adding changesets
adding manifests
adding file changes
added 51849 changesets with 448820 changes to 180327 files


finished applying clone bundle
searching for changes
no changes found
updating to branch default
66512 files updated, 0 files merged, 0 files removed, 0 files unresolved



real	1m59.017s
user	2m10.728s
sys	0m14.323s

[4] ------- remote hgserver, clonebundles enabled, local seed

$ time hg clone http://shipilev.net:8080/ jdk-jdk
applying clone bundle from http://localhost:8000/jdk-jdk.zstd-max.hg
adding changesets
adding manifests
adding file changes
added 51849 changesets with 448820 changes to 180327 files


finished applying clone bundle
searching for changes
no changes found
updating to branch default
66512 files updated, 0 files merged, 0 files removed, 0 files unresolved



real	1m55.545s
user	2m9.893s
sys	0m14.186s


[5] ------- remote git, GitHub mirrors

$ time git clone https://github.com/Project-Skara/jdk.git
Cloning into 'jdk'...
remote: Enumerating objects: 881253, done.
remote: Counting objects: 100% (881253/881253), done.
remote: Compressing objects: 100% (160363/160363), done.
remote: Total 881253 (delta 665729), reused 881253 (delta 665729), pack-reused 0
Receiving objects: 100% (881253/881253), 258.61 MiB | 7.16 MiB/s, done.
Resolving deltas: 100% (665729/665729), done.
Checking out files: 100% (66258/66258), done.

real	1m1.359s
user	0m47.248s
sys	0m6.129s