Cross Component (hotspot+jdk) Development in the Hotspot Group Repos

Tue Sep 9 17:29:06 UTC 2014

There is a README-builds.html file in the top-level repo that has some instructions.

The Adopt OpenJDK project has some good documentation as well: https://java.net/projects/adoptopenjdk/pages/Build

/Staffan

On 9 sep 2014, at 16:36, Coleen Phillimore <coleen.phillimore at oracle.com> wrote:

> 
> Hi,
> 
> Is there a definitive guide on how to build the entire JDK on all platforms that is available on the open mailing lists?
> http://openjdk.java.net/guide/ is all I found.   I don't know how to do this on Windows, for example.
> 
> Thanks,
> Coleen
> 
> On 9/9/14, 2:02 AM, Staffan Larsen wrote:
>> ## tl;dr
>> 
>> We propose a move to a Hotspot development model where we can do both
>> hotspot and jdk changes in the hotspot group repos. This will require a
>> fully populated JDK forest to push changes (whether hotspot or jdk
>> changes) through JPRT. We do not expect these changes to have much
>> affect on the open community, but it is good to note that there can be
>> changes both in hotspot and jdk code coming through the hotspot
>> repositories, and the best practise is to always clone and build the
>> complete forest.
>> 
>> We propose to do this change in a few weeks time.
>> 
>> ## Problem
>> 
>> We see an increasing number of features (small and large) that require
>> concerted changes to both the hotspot and the jdk repos. Our current
>> development model does not support this very well since it requires jdk
>> changes to be made in jdk9/dev and hotspot changes to be made in the
>> hotspot group repositories. Alternatively, such changes results in "flag
>> days" where jdk and hotspot changes are pushed through the group repos
>> with a lot of manual work and impact on everyone working in the group
>> repos. Either way, the result is very slow and cumbersome development.
>> 
>> Some examples where concerted changes have been required are JSR-292,
>> default methods, Java Flight Recorder, work on annotations, moving Class
>> fields to Java, many serviceability area tests, and so on. A lot of this
>> work will continue and we will also see new things such as jigsaw that
>> add to the mix.
>> 
>> Doing concerted changes today takes a lot of manual effort and calendar
>> time to make sure nothing break. In many cases the addition of a new
>> feature needs to made first to a hotspot group repo. That change needs
>> to propagate to jdk9/dev where library code can be changed to depend on
>> it. Once that change has propagated back to the hotspot group repo, the
>> final change can be made to remove the old implementation. This dance
>> can take anywhere from 2 to 4 weeks to complete - for a single feature.
>> 
>> There has also been quite a few cases where we missed taking the
>> dependency into account which results in test failures in one or more
>> repos. In some cases these failures go on for several weeks causing lots
>> of extra work and confusion simply because it takes time for the fix to
>> propagate through the repos.
>> 
>> Instead, we want to move to a model where we can make both jdk and
>> hotspot changes directly in the hotspot group repos. In that way the
>> changes will always "travel together" through the repos. This will make
>> our development cycle faster as well as more reliable.
>> 
>> More or less by definition these type of changes introduce a stronger
>> dependency between hotspot and the jdk. For the product as a whole to
>> work correctly the right combination of hotspot and the jdk need to be
>> used. We have long since removed the requirement that hotspot would
>> support several jdk versions (known as the Hotspot Express - or hsx -
>> model) and we continue to see a strong dependency, where matching code
>> in hotspot and the jdk needs to be used.
>> 
>> ## No More Dependency on Latest Promoted Build
>> 
>> The strong dependency between hotspot and jdk makes it impossible for
>> hotspot to depend on the latest promoted jdk build for testing and
>> development. To elaborate on this; if a change with hotspot+jdk
>> dependencies have been pushed to a group repo, it will not longer be
>> possible to use the latest promoted build for running or testing the
>> version of hotspot built in that repo -- the latest promoted build will
>> not have the latest change to the jdk that hotspot now depends on (or
>> vice versa).
>> 
>> ##  Require Fully Populated JDK Forest
>> 
>> The simple solution that we can switch to today is to always require a
>> fully populated JDK forest when building (both locally and in JPRT). By
>> this we mean a clone of all the repos in the forest under, for example,
>> jdk9/hs-rt. JPRT would no longer be using the latest promoted build when
>> creating bundles, instead it will build the code from the submitted
>> forest.
>> 
>> If all operations (builds, integrations, pushes, JPRT jobs) always work
>> on the full forest, then there will never be a mismatch between the jdk
>> and the hotspot code.
>> 
>> The main drawbacks of this is that developers now need to clone, store
>> and build a lot more code. Cloning the full forest takes longer than
>> just cloning the hotspot forest. This can be alleviated by maintaining
>> local cached versions. Storing full forests require more disk space.
>> This can be mitigated by buying more disks or using a different workflow
>> (for example Mercurial Queues). Building a full jdk takes longer, but
>> hotspot is already one of the larger components to build and incremental
>> builds are usually quite fast.
>> 
>> ## Next Steps
>> 
>> Given that we would like to improve the model we use for cross component
>> development as soon as possible, we would like to switch to require a
>> fully populated JDK forest for hotspot development. All the
>> prerequisites for doing this are in place (changes to JPRT, both on the
>> servers and to the configuration files in the source repos). A group of
>> volunteering hotspot developers have been using full jdk repos for a
>> while for day-to-day work (except pushes) and have not reported any
>> showstopper problems.
>> 
>> If no strong objections are rasied we need decide on a date when we
>> throw the switch. A good date is probably after the 8u40 Feature
>> Complete date of Mid September [0] so as not to impact that release
>> (although this change will only apply to JDK 9 development for now).
>> 
>> Regards,
>> Jon Masamitsu, Karen Kinnear, Mikael Vidstedt,
>> Staffan Larsen, Stefan Särne, Vladimir Kozlov
>> 
>> [0] http://openjdk.java.net/projects/jdk8u/releases/8u40.html
> 