hg clone is unbelievably slow

Tue Feb 6 15:58:22 UTC 2018

On 02/06/2018 04:18 PM, Mario Torre wrote:
> Nice, I didn't know about that extension, but seems exactly what we
> are searching for.

Yep, it fits this use case perfectly.

> What would it take to add this support to the repos?

The clonebundles has to be created periodically (via a cron job, Jenkins 
job or manually), stored somewhere (storing one clonebundle per 
repository would be sufficient IMO) and the server-side configuration 
must be updated (see the documentation for details about that).

As for who can do that, I would ask ops at openjdk.java.net (I'm not part 
of ops).

> From the linked
> documentation I understand this is transparent for clients, so we
> should see immediate improvements just by enabling it on the servers.

Yes, all hg clients that understand clonebundles will use it (and 
thereby get the performance improvement) and older hg clients will just 
continue to use the traditional way of cloning.

Thanks,
Erik

> Cheers,
> Mario
> 
> 2018-02-06 16:04 GMT+01:00 Erik Helin <erik.helin at oracle.com>:
>> There is no need to re-invent the wheel :)
>>
>> Mercurial (hg) has had support for clonebundles for a some time by now, see
>> [0] for details. If the servers where configured to use clonebundles then
>> users can continue to just use `hg clone` but get the performance of
>> downloading compressed bundles. clonebundles can be stored on various CDN
>> servers, then `hg clone` will download the closest bundle, unpack it and
>> eventually do `hg pull --update` to get the latest commits (that might not
>> be present in the bundle).
>>
>> For more information about clonebundles, just run
>> `hg help -e clonebundles`.
>>
>> Thanks,
>> Erik
>>
>> [0]: https://www.mercurial-scm.org/wiki/ClonebundlesExtension
>>
>>
>> On 02/06/2018 03:42 PM, Mario Torre wrote:
>>>
>>> I think it should be a fully functional mercurial repository once is
>>> unpacked.
>>>
>>> What I have in mind is something that support this workflow (or similar):
>>>
>>> $ hg clone  -r 12345 http://hg.openjdk.java.net/jdk10/client-gz
>>> $ tar xvfz client-gz.tgz && cd client-gz
>>> (do whatever, then some time later)
>>> $ hg log || hg pull -u || commit etc...
>>>
>>> This should be just a few lines of scripting away.
>>>
>>> Cheers,
>>> Mario
>>>
>>>
>>>
>>> 2018-02-06 15:35 GMT+01:00 Thomas Stüfe <thomas.stuefe at gmail.com>:
>>>>
>>>> Side question, we would only need to tar the .hg folder, no? And revive
>>>> the
>>>> workspace with hg update?
>>>>
>>>> On Tue, Feb 6, 2018 at 3:10 PM, Mario Torre
>>>> <neugens.limasoftware at gmail.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> I think it should probably be done on demand, it may be more expensive
>>>>> to keep a commit based tgz otherwise.
>>>>>
>>>>> In fact the most awesome thing would be to have a special repository
>>>>> that we clone but gives us back only the tgz to unpack locally, with
>>>>> all the history and correct hg paths.
>>>>>
>>>>> Cheers,
>>>>> Mario
>>>>>
>>>>> 2018-02-06 15:04 GMT+01:00 Remi Forax <forax at univ-mlv.fr>:
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> creating a tgz at each commit doesn't seems that difficult ...
>>>>>>
>>>>>> Rémi
>>>>>>
>>>>>> ----- Mail original -----
>>>>>>>
>>>>>>> De: "Andrew Haley" <aph at redhat.com>
>>>>>>> À: "jdk-dev" <jdk-dev at openjdk.java.net>
>>>>>>> Envoyé: Mardi 6 Février 2018 11:50:07
>>>>>>> Objet: hg clone is unbelievably slow
>>>>>>
>>>>>>
>>>>>>> Half an hour or more here.  AFAIK the problem is due to the
>>>>>>> inefficiency of Mercurial itself and the hg protocol.
>>>>>>>
>>>>>>> Aleksey Shipilev has done an experiment whereby trees are regularly
>>>>>>> cloned and compressed tarballs created; these can be downloaded in a
>>>>>>> couple of minutes.  But really we don't want to depend on the largesse
>>>>>>> of one developer: if we could download the OpenJDK trees directly by
>>>>>>> means of wget (or something similar) we would reduce the load on the
>>>>>>> servers and reduce the time taken to download as well.
>>>>>>>
>>>>>>> --
>>>>>>> Andrew Haley
>>>>>>> Java Platform Lead Engineer
>>>>>>> Red Hat UK Ltd. <https://www.redhat.com>
>>>>>>> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
>>>>> Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF
>>>>>
>>>>> Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
>>>>> Proud GNU Classpath developer: http://www.classpath.org/
>>>>> OpenJDK: http://openjdk.java.net/projects/caciocavallo/
>>>>>
>>>>> Please, support open standards:
>>>>> http://endsoftpatents.org/
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
> 
> 
> 