Process proposal for Updating JDK 7u with Hotspot Express...

[John Coomes]

Erik Trimble (erik.trimble at oracle.com) wrote:
> Hi folks.
> 
> This email is to descibe the the process of integrating Hotspot Express
> versions into the JDK 7u series, and open a comment on the this process.
> It's rather long, so please read it carefully.
> ...

Since Erik has left Oracle, I'll be filling in until a permanent
hotspot gatekeeper is able to take over.

I have some corrections and recommendations below.

> The very latest Hotspot development version is always found here:
> 
> http://hg.openjdk.java.net/hsx/hotspot-main/hotspot
> 
> Current, latest Hotspot version is HSX22.0. A copy of the latest STABLE
> version of HSX22 (i.e. one which has undergone a QA cycle) is found
> here:
> 
> http://hg.openjdk.java.net/hsx/hsx22/hotspot
> 
> Now, normally, after a QA cycle has passed, and the contents of
> hsx22/hotspot are refreshed with the newest stable bits, it is then
> promoted into the OpenJDK 8 repository
> ( hg.openjdk.java.net/jdk8/jdk8/hotspot )

Correction:  the hsxNN/hotspot repository is not guaranteed to be
completely stable; it's a snapshot area used to capture the source
that is about to undergo a PIT (pre-integration test) cycle.  So at
times it will contain changes that have not yet passed PIT.  In
practice it turns out to be stable because HotSpot snapshots have
already undergone one or more nightly testing cycles and historically
have almost always passed PIT.

The basic process is:

	1.  Check the nightly test results for hsx/hotspot-main/ and
	if no significant failures are found, push a snapshot of the
	contents to the current numbered hsx repo, (e.g., hsx/hsx22).
	jprt is used to generate binaries and do the push.

	2. Start a PIT cycle using the jprt-generated binaries,
	testing with the JDK version into which hotspot will be
	integrated (e.g., jdk7u2).  The PIT usually starts on Friday
	and results are available on Tuesday.

	3. While PIT is running, do a complete build of the JDK on all
	platforms using the hotspot source from the snapshot (a
	'control build').  This is also done using jprt, primarily for
	convenience.

	4. If the control build succeeds and the PIT results are
	acceptable (as determined by SQE), integrate the snapshot into
	the target release repository (e.g., into jdk8/jdk8/hotspot or
	jdk7u/jdk7u-dev/hotspot).

> Using the Hotspot Express model in the 7u series involves these steps
> [note that all references to '7u' or '7u-dev' refer to either
> http://hg.openjdk.java.net/jdk7u/jdk7u or
> http://hg.openjdk.java.net/jdk7u/jdk7u-dev, respectively]:
> 
> 1.) Periodically, the contents of the latest hsx/hsxN/hotspot repository
> (i.e. the version matching the one currently in
> hsx/hotspot-main/hotspot ) will be put through a new QA cycle, as a
> candidate for inclusion in the 7u series.
> 
> 2.) To be explicit, the QA cycle using this Hotspot Express snapshot
> will be using the latest 7u JDK, NOT the 8 JDK. Thus, we expect to
> discover any 7-specific issues with Hotspot BEFORE it is pushed into the
> Integration repository (7u-dev).  
> 
> 2a.) In addition, there will never be a Hotspot->7u integration until
> AFTER the same Hotspot version has been promoted into the JDK 8 forest,
> and undergone a full Release promotion cycle. This will be to make sure
> that the Hotspot version in hsx/hsxN/hotspot is indeed stabilized and we
> have worked out any immediately apparent serious issues. This may very
> well mean that Hotspot will not immediately promote all Hotspot builds
> unto 7u.  E.g. HS20 b01 may go to JDK 8 Build 01, but if there are
> problems detected, then HS22 will not be pushed into 7u until those
> issues are addressed in a new HS build. So, it is entirely possible that
> an integration into 7u will actually encompass several Hotspot build
> numbers.

This restriction (2a) will have to be relaxed, at least somewhat.
Since jdk8 builds are not yet happening regularly, requiring the
hotspot snapshot to appear in a promoted jdk8 build before it can be
integrated into 7u will delay 7u integrations for indefinite periods.
So the requirement that a hotspot snapshot complete what Erik calls 'a
full Release promotion cycle' in jdk8 before being integrated into 7u
should be dropped.

In addition, given that build schedules and release dates for 7u and 8
are not aligned, I can easily foresee the case when deadlines will
necessitate integration into 7u before integration into the jdk8
master forest.

So we should change the requirement that a hotspot snapshot be
integrated into the jdk8 *master* before before being integrated into
7u.  Instead, the presence of a fix in *hsx/hotspot-main* should be
enough to meet the requirement that a fix be present in jdk8.  The
hsx/hotspot-main forest is used very much like the jdk8 integration
forests used by other groups (e.g., jdk8/build/*, jdk8/tl/*, etc.), in
that it is regularly pushed up to the jdk8 master.  The only
difference is that it also delivers into other releases.  The process
for approving individual fixes for 7u deems the presence of the fix in
a jdk8 integration forest (e.g., jdk8/tl) as sufficient to meet the
requirement that a fix be included in jdk8; the same should apply to
hotspot.

[More comments below.]

> 3.) At the beginning of this QA cycle, a webrev will be created,
> detailing ALL the changes vs the existing 7u/hotspot repository. That
> is, this will be a very large webrev, as it includes all the fixes
> between the latest development Hotspot and the existing 7u Hotspot.
> 
> 4.) This webrev will be posted to cr.openjdk.java.net, and a "request
> for integration" notice will be sent to jdk7u-dev at openjdk.java.net, as
> normal for other integration requests.
> 
> 5.) The 7u Technical Lead will approve the Hotspot update, taking into
> consideration the timing of the push - that is, approvals should be
> concerned with whether a new Hotspot version is appropriate given build
> schedules, NOT on the technical merits. 
> 
> 6.) After the approval has been received, the Hotspot snapshot
> undergoing QA will be pushed into the 7u-dev/hotspot Integration area.
> 
> 7.) Upon receipt of the QA certification (PIT cert) that Hotspot has
> successfully passed all relevant testing (or, there are only
> inconsequential errors which aren't large enough to warrant a respin),
> the contents of 7u-dev/hotspot will be pushed up to the Master area
> (7u/hotspot ).
> 
> A couple of notes:
> 
> a) I expect that #3 and #4 will happen almost simultaneously. That is,
> our internal snapshot scripts will submit the job to QA and generate the
> webrev together.
> 
> b) #5 should likely happen very shortly after #4, probably the same day.
> 
> c) Frankly, technical discussion of the merits of the Hotspot push is
> not expected to be possible here (in the context of the integration,
> that is). The webrev is an informational post, not a "please review and
> approve" post. This is due to the very large number of CRs being fixed
> (several dozen or more), and the highly complex nature of all of them.
> Such review is carried out when the code is first pushed into
> hsx/hotspot-main and JDK8, so a duplicate review here is not necessary
> and is unlikely to be reasonably possible within the limited timeframe
> that an integration must have. (Translation: we can't post the webrev
> and wait 2 weeks for people to fully review and comment on things before
> integrating it).  
> 
> d) People interested in hotspot development (which will concern use of
> Hotspot in ALL JDK versions, 6, 7, and 8), should subscribe to
> hotspot-dev at openjdk.java.net. 
> 
> e) the 7uN repositories will NOT follow the above procedure - as they
> are "stabilization-only" repositories, Hotspot developers will be
> pushing changes directly to the /hotspot repository under the relevant
> 7uN forest in the same manner as all other stabilization fixes from
> other JDK developers. Thus, all fixes for Hotspot for 7uN releases will
> follow the standard JDK processes.
> 
> One specific point where I'm not sure how we want to proceed is this:
> 
> Should #6 (the push of the Hotspot snapshot into 7u-dev/hotspot) happen
> right after approval from the Technical Lead happen? If so, then it
> likely will be BEFORE QA has finished on the snapshot. This would be in
> line with the other JDK fixes, since they do not undergo QA before being
> pushed to 7u-dev/*.  However, Hotspot is "special", so do we care to be
> extra sure that 7u-dev/hotspot is stable?  
> ...

I think hotspot should complete both a PIT cycle and a control build
of the JDK before we integrate into 7u-dev since (a) we're doing bulk
integrations and the amount of change may be rather large, and (b) we
want to ensure that the hotspot in 7u-dev can be used as a bootstrap
for a JDK build.  We currently don't test the latest hotspot as a
bootstrap to build the JDK as part of our automated nightly or JPRT
tests, so it should be done before integrating into 7u-dev.

We'll follow this order (integrate into jdk7u/jdk7u-dev only after
passing PIT, and into jdk7u/jdk7u shortly afterward) for the current
integration of hs22 b02 into 7u2 b03, expected today.  PIT results are
still pending, but all other requirements have been met.

-John