Cross Component (hotspot+jdk) Development in the Hotspot Group Repos

[Staffan Larsen]

## tl;dr

We propose a move to a Hotspot development model where we can do both
hotspot and jdk changes in the hotspot group repos. This will require a
fully populated JDK forest to push changes (whether hotspot or jdk
changes) through JPRT. We do not expect these changes to have much
affect on the open community, but it is good to note that there can be
changes both in hotspot and jdk code coming through the hotspot
repositories, and the best practise is to always clone and build the
complete forest.

We propose to do this change in a few weeks time.

## Problem

We see an increasing number of features (small and large) that require
concerted changes to both the hotspot and the jdk repos. Our current
development model does not support this very well since it requires jdk
changes to be made in jdk9/dev and hotspot changes to be made in the
hotspot group repositories. Alternatively, such changes results in "flag
days" where jdk and hotspot changes are pushed through the group repos
with a lot of manual work and impact on everyone working in the group
repos. Either way, the result is very slow and cumbersome development.

Some examples where concerted changes have been required are JSR-292,
default methods, Java Flight Recorder, work on annotations, moving Class
fields to Java, many serviceability area tests, and so on. A lot of this
work will continue and we will also see new things such as jigsaw that
add to the mix.

Doing concerted changes today takes a lot of manual effort and calendar
time to make sure nothing break. In many cases the addition of a new
feature needs to made first to a hotspot group repo. That change needs
to propagate to jdk9/dev where library code can be changed to depend on
it. Once that change has propagated back to the hotspot group repo, the
final change can be made to remove the old implementation. This dance
can take anywhere from 2 to 4 weeks to complete - for a single feature.

There has also been quite a few cases where we missed taking the
dependency into account which results in test failures in one or more
repos. In some cases these failures go on for several weeks causing lots
of extra work and confusion simply because it takes time for the fix to
propagate through the repos.

Instead, we want to move to a model where we can make both jdk and
hotspot changes directly in the hotspot group repos. In that way the
changes will always "travel together" through the repos. This will make
our development cycle faster as well as more reliable.

More or less by definition these type of changes introduce a stronger
dependency between hotspot and the jdk. For the product as a whole to
work correctly the right combination of hotspot and the jdk need to be
used. We have long since removed the requirement that hotspot would
support several jdk versions (known as the Hotspot Express - or hsx -
model) and we continue to see a strong dependency, where matching code
in hotspot and the jdk needs to be used.

## No More Dependency on Latest Promoted Build

The strong dependency between hotspot and jdk makes it impossible for
hotspot to depend on the latest promoted jdk build for testing and
development. To elaborate on this; if a change with hotspot+jdk
dependencies have been pushed to a group repo, it will not longer be
possible to use the latest promoted build for running or testing the
version of hotspot built in that repo -- the latest promoted build will
not have the latest change to the jdk that hotspot now depends on (or
vice versa).

##  Require Fully Populated JDK Forest

The simple solution that we can switch to today is to always require a
fully populated JDK forest when building (both locally and in JPRT). By
this we mean a clone of all the repos in the forest under, for example,
jdk9/hs-rt. JPRT would no longer be using the latest promoted build when
creating bundles, instead it will build the code from the submitted
forest.

If all operations (builds, integrations, pushes, JPRT jobs) always work
on the full forest, then there will never be a mismatch between the jdk
and the hotspot code.

The main drawbacks of this is that developers now need to clone, store
and build a lot more code. Cloning the full forest takes longer than
just cloning the hotspot forest. This can be alleviated by maintaining
local cached versions. Storing full forests require more disk space.
This can be mitigated by buying more disks or using a different workflow
(for example Mercurial Queues). Building a full jdk takes longer, but
hotspot is already one of the larger components to build and incremental
builds are usually quite fast.

## Next Steps

Given that we would like to improve the model we use for cross component
development as soon as possible, we would like to switch to require a
fully populated JDK forest for hotspot development. All the
prerequisites for doing this are in place (changes to JPRT, both on the
servers and to the configuration files in the source repos). A group of
volunteering hotspot developers have been using full jdk repos for a
while for day-to-day work (except pushes) and have not reported any
showstopper problems.

If no strong objections are rasied we need decide on a date when we
throw the switch. A good date is probably after the 8u40 Feature
Complete date of Mid September [0] so as not to impact that release
(although this change will only apply to JDK 9 development for now).

Regards,
Jon Masamitsu, Karen Kinnear, Mikael Vidstedt,
Staffan Larsen, Stefan Särne, Vladimir Kozlov

[0] http://openjdk.java.net/projects/jdk8u/releases/8u40.html