Update on JDK 10 repo consolidation; third generation prototype published at http://hg.openjdk.java.net/jdk10/consol-proto

Fri Sep 8 11:43:34 UTC 2017

As a comparison, FWIW, I tested with my computer at home (nominally 250 
Mbit/s). I cloned the consolidated repo in ~33 minutes, which 
corresponds to an average speed of 800 kB/s or rougly 6 Mbit/s), given 
that all that time was spent transmitting data (and that I've done the 
calculations right).

That sounds like a transfer rate that's close to what I'd expect of an 
arbitrary, non-speed-optimized, service on the Internet in general. (On 
the other hand, one might argue that one would expect more of a service 
like hg.openjdk.java.net.)

/Magnus

On 2017-09-07 12:23, Volker Simonis wrote:
> Yes, I can also confirm Thomas' numbers. Tried from home on my 25Mbps
> line with both company/private PC and couldn't get more than 350KB
> download speed during cloning.
>
> I've also tested a traditional repository with the get_source.sh
> script. By default the script uses a concurrency of two (i.e. it
> always processes two repositories of the forest in parallel). This
> obviously starts two hg processes at a time and it seems that both of
> them can use up to 350KB during downloading. This gets the cloning
> time down to 12m (compared to 38m for the new, fat consolidated
> repository).
>
> The concurrency of get_source.sh can be increased by setting
> HGFOREST_CONCURRENCY. If I set this to 8 the cloning time only drops
> down to 11 minutes because it is bound by the time it takes to clone
> the jdk repository which is by far the biggest one.
>
> So it really seems that the hg.openjdk.java.net limits the download
> speed per connection. Before, with the forest, this was not so much of
> an issue, because cloning was done in parallel for each forest, but
> now, with the new, single, fat, consolidated repository, this becomes
> a real bottleneck.
>
> Can somebody from Oracle and/or the hg.openjdk.java.net operations
> team please comment on this and ideally fix it?
>
> Thank you and best regards,
> Volker
>
>
> On Thu, Sep 7, 2017 at 10:07 AM, Doug Simon <doug.simon at oracle.com> wrote:
>>> On 7 Sep 2017, at 07:06, Thomas Stüfe <thomas.stuefe at gmail.com> wrote:
>>>
>>> On Wed, Sep 6, 2017 at 5:35 PM, Volker Simonis <volker.simonis at gmail.com>
>>> wrote:
>>>
>>>> On Wed, Sep 6, 2017 at 2:30 PM, Thomas Stüfe <thomas.stuefe at gmail.com>
>>>> wrote:
>>>>> Erik, Volker,
>>>>>
>>>>> On Tue, Sep 5, 2017 at 7:31 PM, Erik Joelsson <erik.joelsson at oracle.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> On 2017-09-05 11:38, Volker Simonis wrote:
>>>>>>> Hi Joe,
>>>>>>>
>>>>>>> generally looks good!
>>>>>>>
>>>>>>> I don't know if this has discussed before, but in my opinion the
>>>>>>> top-level src/ directory looks a little overloaded. Wouldn't it make
>>>>>>> sense to place all the modules into their own 'src/modules/'
>>>>>>> subdirectory?
>>>>>> I see what you mean, but it's no more overloaded than the jdk/src dir
>>>> used
>>>>>> to be. Also, the bsd, linux, solaris, demo and sample directories are
>>>> all
>>>>>> going away at some point in the (hopefully near) future. Left are
>>>> hotspot
>>>>>> and utils which I don't think warrant another directory level.
>>>>>>> Should jdk10/consol-proto build out of t he box or are there any known
>>>>>>> issues? I'd like to give it a tray on AIX and Linux/ppc64 but if there
>>>>>>> are any known, generic problems I'll wait until they get fixed.
>>>>>> Yes, please try it. I have put a lot of work into maintaining a set of
>>>>>> patches and scripts to keep the generated consolidated repo working and
>>>>>> (near) equivalent with the current forest. You can review those changes
>>>> in
>>>>>> the prototype forest if you like.
>>>>>>
>>>>> I cloned http://hg.openjdk.java.net/jdk10/consol-proto and built AIX.
>>>> Build
>>>>> runs fine.
>>>>>
>>>> Hi Thomas,
>>>>
>>>> thanks for checking the AIX build.
>>>>
>>>>> However, I found cloning the repository painfully slow.
>>>>>
>>>>> On my Linux box clone took ~90 minutes. On Windows, I could not
>>>> successfully
>>>>> clone, as cygwin mercurial did hit timeouts. I really hope this is only
>>>>> temporary and will improve?
>>>>>
>>>> I think you should ask your company and/or Internet provider :)
>>>>
>>>> The new, consolidated repository is about 1.6GB in size (i.e. the size
>>>> of .hg) and the cloning speed is actually pretty much proportional to
>>>> the time you need to download this amount of data.
>>>>
>>>> When using our http-proxy I measured a download speed of about 350
>>>> KB/s. The calculation is quite simple:
>>>>
>>>> 1600000 KB / 350 KB/s = 4571 s / 60 = 76 min :(
>>>>
>>>> The actual cloning speed was slightly faster:
>>>>
>>>> $ time hg clone http://hg.openjdk.java.net/jdk10/consol-proto
>>>> jdk10-cons-proto
>>>> requesting all changes
>>>> adding changesets
>>>> adding manifests
>>>> adding file changes
>>>> added 46937 changesets with 397578 changes to 162338 files
>>>> updating to branch default
>>>> 57067 files updated, 0 files merged, 0 files removed, 0 files
>>>> unresolved
>>>>
>>>> real    38m37.424s
>>>> user    5m46.620s
>>>> sys    0m33.976s
>>>>
>>>> which is probably because the 1.6 GB from the .hg repo may be
>>>> compressed for the transport.
>>>>
>>>> Without proxy (i.e. 'transparent proxy') the average download speed is
>>>> about 150 KB/s. I think this pretty much explains the ~90 minutes.
>>>>
>>>> I doubt that these poor download speeds are caused by
>>>> hg.openjdk.java.net because then it would be the same for the proxy
>>>> vs. non-proxy case. Nevertheless it would be interesting to see what
>>>> amount of data hg.openjdk.java.net can actually serve to Europe. So if
>>>> somebody with a decent Internet connection can share his experience,
>>>> that would be interesting.
>>>>
>>>>
>>> I'm on cable, with a download speed of 128Mbit. Cloning the new repo still
>>> took 39min. That was on Linux, with SSDs.
>> I'm in Switzerland, get 85Mbps according to fast.com and it took 42min to clone the new repo.
>>
>> -Doug
>>
>>> Best Regards, Thomas
>>>
>>>
>>>> Regards,
>>>> Volker
>>>>
>>>>> Kind Regards, Thomas
>>>>>
>>>>>
>>>>>> /Erik
>>>>>>
>>>>>>> Thank you and best regards,
>>>>>>> Volker
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 1, 2017 at 8:36 PM, joe darcy <joe.darcy at oracle.com>
>>>> wrote:
>>>>>>>> On 8/25/2017 1:52 PM, joe darcy wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> A follow-up to the most recent update [1] on the JDK 10 repo
>>>>>>>>> consolidation
>>>>>>>>> efforts, September 2017 is almost upon us and that month remains the
>>>>>>>>> target
>>>>>>>>> to implement the repo consolidation.
>>>>>>>>>
>>>>>>>>> First, a third generation prototype having tags from both JDK 9 and
>>>> JDK
>>>>>>>>> 10
>>>>>>>>> will be published in the near future.
>>>>>>>>>
>>>>>>>> [snip]
>>>>>>>>
>>>>>>>> Third generation prototype with tags from JDK 9 and JDK 10 available
>>>> for
>>>>>>>> browsing at:
>>>>>>>>
>>>>>>>>      http://hg.openjdk.java.net/jdk10/consol-proto/tags
>>>>>>>>
>>>>>>>> Please sent comments by Wednesday, September 6.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> -Joe