Looking ahead: proposed Hg forest consolidation for JDK 10

[Jonathan Gibbons]

On 10/12/2016 03:04 PM, Mikael Vidstedt wrote:
>> On Oct 12, 2016, at 9:06 AM, Andrew Hughes <gnu.andrew at redhat.com> wrote:
>>
>> snip...
>>
>>>>> I would appreciate if the runner could be included in the
>>>>> root/test directory.
>>>> I'm not quite sure what you are referring to by the jtreg runner.
>>> I mean the code in http://hg.openjdk.java.net/code-tools/jtreg
>>>
>> I think the problem with that is that the development of jtreg then becomes
>> tied to OpenJDK. It might be ok for the development version of OpenJDK, but
>> you then have to backport jtreg fixes into update releases, rather than just
>> using the same jtreg from the separate repository.
>>
>> We've experienced that bundling issue with IcedTea, and it's why we split
>> out the plugin/javaws work (IcedTea-Web).
>>
>>> As Andrew stated, some subdirectories are pretty stable. It
>>> might completely make sense to merge these into one repository, but I'm
>>> really concerned about jdk and hotspot.
>>>
>>> In general, I think those people that are highly specialized on complex
>>> subcomponents of the VM will suffer from this.  They often are fine
>>> just working with hotspot / jdk etc..  In general, these people develop
>>> new components in the latest branch.
>>> Those people that have to maintain and test the VM really will profit
>>> from the new setup.  They anyways always operate with the full
>>> repo tree.
>>> Having this said, I think it would make more sense to put the legacy code
>>> base into merged repos, and not the development branch?
>>>
>> I agree the biggest friction is going to be between HotSpot and the other
>> repositories, much as it was for the build system changes which took an
>> entire OpenJDK release to make it to HotSpot, because the development
>> experience is quite different.
>>
>> The HotSpot repository is still fairly slim and performs well. I get the
>> impression that many HotSpot developers work on that repository in isolation,
>> given their requirements to be able to build from within the repo rather than
>> the top-level.
>
> FWIW since a few years ago all HotSpot developers at Oracle are effectively required to pull and work with the full set of repos. At the time of the switchover, developers had to spend some time adapting workflows and (wrapper) build scripts, etc. to the new model, but I think it’s fair to say by now that it has been working out really well and that it has over the years meant that many events which would otherwise have been “flag days” (where incompatible changes need to be made across the JDK/HotSpot boundary), have just been handled like a normal integrations/changes. Extrapolating from lessons learned in the past we have definitely saved ourselves a lot of trouble by not treating the hotspot repo/code as a special case. Somewhat naively I actually think that it also leads to better design and implementation choices, because you don’t have to think twice about making that breaking cross-repo change - if that is the right way to implement it, that is what you do, no need to worry about breaking your colleagues’ builds.
>
>> The JDK repository is already experiencing slowdown. It doesn't really work
>> without the top-level repository and the additional components (CORBA, JAXP,
>> JAXWS, Nashorn) are usually needed for a build, even if they are seldom altered.
>> Given the size of the JDK repository already, adding in these components is
>> not going to make a huge difference.
>>
>> Build changes usually end up touching most of the repositories. Merges certainly
>> do. So, yes, the benefits are greatest for those doing build and integration work.
> Not to mention big, cross-component projects like Lambdas and Jigsaw, projects which are complex enough even without taking repositories etc. into account.
>
>> I do wonder what percentage of the cross-repo commits touch HotSpot, and whether
>> we could retain that as a separate repository if it's going to cause so much
>> disruption for HS developers.
>>
>> JDK builds can be done using an imported HotSpot and HotSpot developers can use an
>> imported JDK, so I don't see a strong reason to join these together. We've had to
>> swap out HotSpot with one that includes a newer port on several occasions, and
>> having to duplicate all the JDK code in these cases would be a major pain.
> Even after a potential repo consolidation I would expect it to be possible to only (re-)build the hotspot component, without having to necessarily recompile the rest of the JDK (the current build system already supports this). It may also be possible to import/export it into an existing JDK, but it should be noted that we do not in any way guarantee that the end result will work as intended unless the code for the pre-built JDK matches the Hotspot code. As Joe mentions, we have made several potentially breaking changes like that over the last few years - I did at least a couple of breaking changes myself related to the Unsafe cleanup.

Ditto from LangTools.  Currently, those of us working on the javac and 
javadoc (in particular) can use a simple Ant script to (re)build just 
the modules in the langtools/ repo. Yes, it's do at your own risk, and 
not guaranteed to work, and requires maintenance, but I would expect 
that such mechanisms will continue to work after the consolidation.

>
> Cheers,
> Mikael
>
>> HotSpot also operates a different commit process which could cause friction if
>> the repos are merged.
>>
>>> Best regards,
>>>   Goetz.
>>>
>>>
>>>
>>>
>> -- 
>> Andrew :)
>>
>> Senior Free Java Software Engineer
>> Red Hat, Inc. (http://www.redhat.com)
>>
>> PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net)
>> Fingerprint = 5132 579D D154 0ED2 3E04  C5A0 CFDA 0F9B 3596 4222
>>
>>