Update on JDK 10 repo consolidation; third generation prototype published at http://hg.openjdk.java.net/jdk10/consol-proto

[Volker Simonis]

On Wed, Sep 6, 2017 at 2:30 PM, Thomas Stüfe <thomas.stuefe at gmail.com> wrote:
> Erik, Volker,
>
> On Tue, Sep 5, 2017 at 7:31 PM, Erik Joelsson <erik.joelsson at oracle.com>
> wrote:
>>
>>
>>
>> On 2017-09-05 11:38, Volker Simonis wrote:
>>>
>>> Hi Joe,
>>>
>>> generally looks good!
>>>
>>> I don't know if this has discussed before, but in my opinion the
>>> top-level src/ directory looks a little overloaded. Wouldn't it make
>>> sense to place all the modules into their own 'src/modules/'
>>> subdirectory?
>>
>> I see what you mean, but it's no more overloaded than the jdk/src dir used
>> to be. Also, the bsd, linux, solaris, demo and sample directories are all
>> going away at some point in the (hopefully near) future. Left are hotspot
>> and utils which I don't think warrant another directory level.
>>>
>>> Should jdk10/consol-proto build out of t he box or are there any known
>>> issues? I'd like to give it a tray on AIX and Linux/ppc64 but if there
>>> are any known, generic problems I'll wait until they get fixed.
>>
>> Yes, please try it. I have put a lot of work into maintaining a set of
>> patches and scripts to keep the generated consolidated repo working and
>> (near) equivalent with the current forest. You can review those changes in
>> the prototype forest if you like.
>>
>
> I cloned http://hg.openjdk.java.net/jdk10/consol-proto and built AIX. Build
> runs fine.
>

Hi Thomas,

thanks for checking the AIX build.

> However, I found cloning the repository painfully slow.
>
> On my Linux box clone took ~90 minutes. On Windows, I could not successfully
> clone, as cygwin mercurial did hit timeouts. I really hope this is only
> temporary and will improve?
>

I think you should ask your company and/or Internet provider :)

The new, consolidated repository is about 1.6GB in size (i.e. the size
of .hg) and the cloning speed is actually pretty much proportional to
the time you need to download this amount of data.

When using our http-proxy I measured a download speed of about 350
KB/s. The calculation is quite simple:

1600000 KB / 350 KB/s = 4571 s / 60 = 76 min :(

The actual cloning speed was slightly faster:

$ time hg clone http://hg.openjdk.java.net/jdk10/consol-proto jdk10-cons-proto
requesting all changes
adding changesets
adding manifests
adding file changes
added 46937 changesets with 397578 changes to 162338 files
updating to branch default
57067 files updated, 0 files merged, 0 files removed, 0 files
unresolved

real    38m37.424s
user    5m46.620s
sys    0m33.976s

which is probably because the 1.6 GB from the .hg repo may be
compressed for the transport.

Without proxy (i.e. 'transparent proxy') the average download speed is
about 150 KB/s. I think this pretty much explains the ~90 minutes.

I doubt that these poor download speeds are caused by
hg.openjdk.java.net because then it would be the same for the proxy
vs. non-proxy case. Nevertheless it would be interesting to see what
amount of data hg.openjdk.java.net can actually serve to Europe. So if
somebody with a decent Internet connection can share his experience,
that would be interesting.

Regards,
Volker

> Kind Regards, Thomas
>
>
>>
>> /Erik
>>
>>>
>>> Thank you and best regards,
>>> Volker
>>>
>>>
>>> On Fri, Sep 1, 2017 at 8:36 PM, joe darcy <joe.darcy at oracle.com> wrote:
>>>>
>>>> On 8/25/2017 1:52 PM, joe darcy wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> A follow-up to the most recent update [1] on the JDK 10 repo
>>>>> consolidation
>>>>> efforts, September 2017 is almost upon us and that month remains the
>>>>> target
>>>>> to implement the repo consolidation.
>>>>>
>>>>> First, a third generation prototype having tags from both JDK 9 and JDK
>>>>> 10
>>>>> will be published in the near future.
>>>>>
>>>> [snip]
>>>>
>>>> Third generation prototype with tags from JDK 9 and JDK 10 available for
>>>> browsing at:
>>>>
>>>>      http://hg.openjdk.java.net/jdk10/consol-proto/tags
>>>>
>>>> Please sent comments by Wednesday, September 6.
>>>>
>>>> Thanks,
>>>>
>>>> -Joe
>>
>>
>