Update on JDK 10 repo consolidation; third generation prototype published at http://hg.openjdk.java.net/jdk10/consol-proto

Thu Sep 7 10:23:56 UTC 2017

Yes, I can also confirm Thomas' numbers. Tried from home on my 25Mbps
line with both company/private PC and couldn't get more than 350KB
download speed during cloning.

I've also tested a traditional repository with the get_source.sh
script. By default the script uses a concurrency of two (i.e. it
always processes two repositories of the forest in parallel). This
obviously starts two hg processes at a time and it seems that both of
them can use up to 350KB during downloading. This gets the cloning
time down to 12m (compared to 38m for the new, fat consolidated
repository).

The concurrency of get_source.sh can be increased by setting
HGFOREST_CONCURRENCY. If I set this to 8 the cloning time only drops
down to 11 minutes because it is bound by the time it takes to clone
the jdk repository which is by far the biggest one.

So it really seems that the hg.openjdk.java.net limits the download
speed per connection. Before, with the forest, this was not so much of
an issue, because cloning was done in parallel for each forest, but
now, with the new, single, fat, consolidated repository, this becomes
a real bottleneck.

Can somebody from Oracle and/or the hg.openjdk.java.net operations
team please comment on this and ideally fix it?

Thank you and best regards,
Volker

On Thu, Sep 7, 2017 at 10:07 AM, Doug Simon <doug.simon at oracle.com> wrote:
>
>> On 7 Sep 2017, at 07:06, Thomas Stüfe <thomas.stuefe at gmail.com> wrote:
>>
>> On Wed, Sep 6, 2017 at 5:35 PM, Volker Simonis <volker.simonis at gmail.com>
>> wrote:
>>
>>> On Wed, Sep 6, 2017 at 2:30 PM, Thomas Stüfe <thomas.stuefe at gmail.com>
>>> wrote:
>>>> Erik, Volker,
>>>>
>>>> On Tue, Sep 5, 2017 at 7:31 PM, Erik Joelsson <erik.joelsson at oracle.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2017-09-05 11:38, Volker Simonis wrote:
>>>>>>
>>>>>> Hi Joe,
>>>>>>
>>>>>> generally looks good!
>>>>>>
>>>>>> I don't know if this has discussed before, but in my opinion the
>>>>>> top-level src/ directory looks a little overloaded. Wouldn't it make
>>>>>> sense to place all the modules into their own 'src/modules/'
>>>>>> subdirectory?
>>>>>
>>>>> I see what you mean, but it's no more overloaded than the jdk/src dir
>>> used
>>>>> to be. Also, the bsd, linux, solaris, demo and sample directories are
>>> all
>>>>> going away at some point in the (hopefully near) future. Left are
>>> hotspot
>>>>> and utils which I don't think warrant another directory level.
>>>>>>
>>>>>> Should jdk10/consol-proto build out of t he box or are there any known
>>>>>> issues? I'd like to give it a tray on AIX and Linux/ppc64 but if there
>>>>>> are any known, generic problems I'll wait until they get fixed.
>>>>>
>>>>> Yes, please try it. I have put a lot of work into maintaining a set of
>>>>> patches and scripts to keep the generated consolidated repo working and
>>>>> (near) equivalent with the current forest. You can review those changes
>>> in
>>>>> the prototype forest if you like.
>>>>>
>>>>
>>>> I cloned http://hg.openjdk.java.net/jdk10/consol-proto and built AIX.
>>> Build
>>>> runs fine.
>>>>
>>>
>>> Hi Thomas,
>>>
>>> thanks for checking the AIX build.
>>>
>>>> However, I found cloning the repository painfully slow.
>>>>
>>>> On my Linux box clone took ~90 minutes. On Windows, I could not
>>> successfully
>>>> clone, as cygwin mercurial did hit timeouts. I really hope this is only
>>>> temporary and will improve?
>>>>
>>>
>>> I think you should ask your company and/or Internet provider :)
>>>
>>> The new, consolidated repository is about 1.6GB in size (i.e. the size
>>> of .hg) and the cloning speed is actually pretty much proportional to
>>> the time you need to download this amount of data.
>>>
>>> When using our http-proxy I measured a download speed of about 350
>>> KB/s. The calculation is quite simple:
>>>
>>> 1600000 KB / 350 KB/s = 4571 s / 60 = 76 min :(
>>>
>>> The actual cloning speed was slightly faster:
>>>
>>> $ time hg clone http://hg.openjdk.java.net/jdk10/consol-proto
>>> jdk10-cons-proto
>>> requesting all changes
>>> adding changesets
>>> adding manifests
>>> adding file changes
>>> added 46937 changesets with 397578 changes to 162338 files
>>> updating to branch default
>>> 57067 files updated, 0 files merged, 0 files removed, 0 files
>>> unresolved
>>>
>>> real    38m37.424s
>>> user    5m46.620s
>>> sys    0m33.976s
>>>
>>> which is probably because the 1.6 GB from the .hg repo may be
>>> compressed for the transport.
>>>
>>> Without proxy (i.e. 'transparent proxy') the average download speed is
>>> about 150 KB/s. I think this pretty much explains the ~90 minutes.
>>>
>>> I doubt that these poor download speeds are caused by
>>> hg.openjdk.java.net because then it would be the same for the proxy
>>> vs. non-proxy case. Nevertheless it would be interesting to see what
>>> amount of data hg.openjdk.java.net can actually serve to Europe. So if
>>> somebody with a decent Internet connection can share his experience,
>>> that would be interesting.
>>>
>>>
>> I'm on cable, with a download speed of 128Mbit. Cloning the new repo still
>> took 39min. That was on Linux, with SSDs.
>
> I'm in Switzerland, get 85Mbps according to fast.com and it took 42min to clone the new repo.
>
> -Doug
>
>>
>> Best Regards, Thomas
>>
>>
>>> Regards,
>>> Volker
>>>
>>>> Kind Regards, Thomas
>>>>
>>>>
>>>>>
>>>>> /Erik
>>>>>
>>>>>>
>>>>>> Thank you and best regards,
>>>>>> Volker
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 1, 2017 at 8:36 PM, joe darcy <joe.darcy at oracle.com>
>>> wrote:
>>>>>>>
>>>>>>> On 8/25/2017 1:52 PM, joe darcy wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> A follow-up to the most recent update [1] on the JDK 10 repo
>>>>>>>> consolidation
>>>>>>>> efforts, September 2017 is almost upon us and that month remains the
>>>>>>>> target
>>>>>>>> to implement the repo consolidation.
>>>>>>>>
>>>>>>>> First, a third generation prototype having tags from both JDK 9 and
>>> JDK
>>>>>>>> 10
>>>>>>>> will be published in the near future.
>>>>>>>>
>>>>>>> [snip]
>>>>>>>
>>>>>>> Third generation prototype with tags from JDK 9 and JDK 10 available
>>> for
>>>>>>> browsing at:
>>>>>>>
>>>>>>>     http://hg.openjdk.java.net/jdk10/consol-proto/tags
>>>>>>>
>>>>>>> Please sent comments by Wednesday, September 6.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> -Joe
>