new convention for hsx project repository names

[John Coomes]

Volker Simonis (volker.simonis at gmail.com) wrote:

Hi Volker,

It's taken a while for me to get back to this; see my responses inline
below.

> Hi John,
> 
> HotSpot versioning has always been a kind of mystery. I'll try to explain my
> understanding of if before I'll come to my question:
> ...
> The easy part is the "hsx/hotspot-main" repository. It contains the linear,
> incremental development code line of the HotSpot. For convenience, larger
> development frames are consecutively numbered with HotSpot version numbers
> (i.e. hs23, hs24, etc.). Once a major development period is over (and before
> the next one begins) the HotSpot may be branched into a stabilization
> repository which until now is named according to the corresponding HotSpot
> version number (i.e. hs23 -> hsx23).
> 
> As the name  "stabilization repositories" implies, they are used to stabilize a
> HotSpot version for a release. Therefore changes from the main development code
> line are selectively integrated into the hsxXY repositories (indicated by the
> '+' sign in my drawing) as needed. Actually, the general rule of thumb is that
> all the changes have to go trough the development code line (i.e. "hsx/
> hotspot-main") before they can be integrated into a "hsxXY" repository although
> it already happened in the past that this rule has been violated. At some point
> in time, hsxXY is considered stable and will be shipped inside a Java release 
> - say jdk7uMN.

Yes, the above is correct.  FWIW, the fixes that don't go through
hsx/hotspot-main first are either not applicable (because the
hotspot-main code has diverged), or critical items which would miss a
deadline (or the release) if we had to wait for it to go to
hotspot-main.  We really dislike the latter.

> But things get more complicated: at a certain point in time, there may be a
> need to stabilize multiple hsxXY versions for different releases in parallel
> (e.g. the next feature and the next security release or the next security
> release and an exceptional security release). In such a case, new hsx
> repositories are created with an additional, minor version number (e.g.
> hsx23.1, hsx23.2).
> 
> At this point things already get nebulous, because the relation of two hsx
> versions hsx.XY.A and hsxXY.B is not always clear to me - i.e. it is not always
> easy to see from which common ancestor change set they are branched.

Yes, it can be hard to find the common ancestor.  But the invariant is
that hsx.XY.B was originally cloned from hsx.XY.A.  There can be
occasional merges from hsx.XY.A into hsx.XY.B, and a final merge when
hsx.XY.A ships (or soon after).

> Another problem is that within the OpenJDK we usually don't see all the
> different hsx.XY.A repositories (some like the ones for security fixes are
> Oracle-internal). Instead, they are all merged into the common hsx.XY
> repository after they have been released.

Right; these are primarily the security releases, which we cannot push
into the open until the release has shipped.

> So far that's how I understand the current situation (comments and corrections
> always welcome:).

You've reverse-engineered the process quite well.

> Now my questions:
> 
> 1. Will you still maintain the major and minor HotSpot versions numbers in make
> /hotspot_version after branching the HotSpot sources to hsx/jdkXu and hsx/
> jdkXuYZ

To keep the user-visible changes in jdk7 updates to a minimum, we will
continue to use hsx major & minor version numbers there.  However, we
want to stop using them at some point, possibly as soon as jdk 8 (but
that will be a stretch).

> 2. What happens if hs26 will go into jdk8uXY. From my understanding of your
> proposal we will end up with hs26 being in hsx/jdk9 AND hsx/jdk8u AND hsx/
> jdk8uXY AND any other  hsx/jdk8uNM with NM > XY. Morover hsx/jdk8u will switch
> from hsx25 to hsx26.

If we did that, once we get to the point of needing a hsx/jdk8uXY
stabilization tree, we would change hsx versions, since the sources
will diverge.  For example, jdk8uXY would stay hs26, and
hsx/hotspot-main would become hs27.  There are several possible
variants depending upon how long we wanted to continue delivering
hsx/hotspot-main into hsx/jdk8u.  But a lot of this should be
unnecessary, see below.

> 3. What happens with the repositories which are not publicly visible? Will they
> all be merged into hsx/jdk8u? We then may get the strange situation that we
> merged hs26 into hsx/jdk8u because it will be backported into jdk8. Later on,
> but before a jdk8 with hs26 will be released, we merge hs25.N (from a security
> release) into hsx/jdk8u. This seems much more confusing than having distinct
> hsx/hs25.N and hsx/hs26.M repositories.

We've been doing the above in hs24/jdk7u for the last year, or more.
For example, security release 7u25 (hs23.25) was merged into hs24 in
hs24-b55; 7u21 (hs23.21) was also merged.  Yes, the merges are a bit
ugly, but are the lowest cost method of ensuring that all security
fixes make it into later releases.

That part of the process doesn't change, except for the naming, and it
should be easier to follow.  After it ships, jdk8u/jdk8u<N> (a
security release) would be merged into jdk8u/jdk8u, and eventually
make its way into hsx/jdk8u.  Fortunately, only the update releases
have these merges; we won't see them in the hsx/hotspot-main repo.

> 4. Why not just stay with the current scheme and just provide a list with the
> hs<NM> to jdkXu<YZ> mapping at a prominent place?

We've maintained such a list internally, and it was still confusing to
people who are responsible for tracking what's in releases, as well as
to new developers and to those who work only occasionally with
hotspot.  It's just an extra step that can be eliminated.

> In my opinion, your scheme will probably always cause trouble because there is
> no one-to-one relation between the HotSpot and JDK version. It would be hard to
> guarantee this because there might be always good reasons to downport a newer
> HotSpot version into an already released JDK.

We've (usually) had a 1-1 relation once we reach the point of needing
a stabilization forest.  And in practice, hotspot has been 1-1 for
more than a year--hs25 has been delivered only to jdk8, and hs24 only
to jdk7u40 during that time.  As Staffan mentioned, that's a trend we
expect to continue.  There is less pressure to deliver every bug fix
and every feature into old JDK releases.  We may still backport many
or even most fixes, but fewer features.

There's also the cost of keeping hotspot compatible with multiple JDK
releases, which requires that we support the associated old O/S
versions and old tool chains.  There are some crusty workarounds in
the code that would be nice to remove.  Larger features like jsr292
and native metadata (aka perm gen removal) would be painful to keep
compatible, and involve more risk than is wanted for update releases,
so it made the most sense to diverge for them.  We want to make larger
features like those easier to do in hotspot.

-John

> On Sat, Aug 31, 2013 at 12:32 AM, John Coomes <John.Coomes at oracle.com> wrote:
> 
>     David Holmes (david.holmes at oracle.com) wrote:
>     > Hi John,
>     >
>     > Can you clarify how the repository name change will affect the actual
>     > hotspot version info that is maintained in the repo, and reported via
>     > -version and -Xinternalversion.
> 
>     Hi David,
> 
>     The new convention only affects the repository names; the version
>     reported by the JVM will remain the same for now.
> 
>     Changing the version string is an obvious future step, but since it's
>     visible to users (not just developers), we have to be a bit more
>     careful.  Look for more email on that in the coming weeks.
> 
>     -John
> 
>     > On 30/08/2013 2:09 PM, John Coomes wrote:
>     > > The hsx Project has maintained a version number for HotSpot that is
>     > > distinct from the JDK version--for example HotSpot version hs24 is
>     > > being delivered into jdk7u40, and hs25 into jdk8.  (For interested
>     > > readers, some background info about separate versions is included at
>     > > the end of this message.)
>     > >
>     > > The separate version has also been reflected in repository paths, e.g.:
>     > >
>     > >     http://hg.openjdk.java.net/hsx/hsx24
>     > >     http://hg.openjdk.java.net/hsx/hsx23
>     > >     http://hg.openjdk.java.net/hsx/hsx23.2
>     > >     http://hg.openjdk.java.net/hsx/hsx23.4
>     > >     ...
>     > >
>     > > One often-mentioned problem with this naming scheme is that the
>     > > correspondence between an hsx repository and the JDK release into which
>     > > it will be delivered is not obvious.  So we are planning on changing
>     the
>     > > repository naming convention going forward.
>     > >
>     > > More precisely, the new repository for HotSpot (and HotSpot-related)
>     > > changes targeting jdk7u60 and later 7 updates would be:
>     > >
>     > >     http://hg.openjdk.java.net/hsx/jdk7u
>     > >
>     > > (The old convention would have used the name http://.../hsx/hsx24.<N>)
>     > > This new repo will correspond to the jdk7u on-going development repo,
>     > > i.e., http://hg.openjdk.java.net/jdk7u/jdk7u.
>     > >
>     > > Extending this a little into the future, once jdk7u60 reaches a point
>     > > where a separate stabilization repo is needed, we will create
>     > > http://hg.openjdk.java.net/hsx/jdk7u60.  At that time the hsx/jdk7u
>     > > repo would change to target the following jdk7u update release (if
>     > > there is one).
>     > >
>     > > Similar conventions would apply to repositories for jdk8 updates once
>     > > we have a need for them.
>     > >
>     > > Existing hsx repos should continue to be used; in particular, we will
>     > > continue to use the on-going development repos for the forseeable
>     > > future:
>     > >
>     > >     http://hg.openjdk.java.net/hsx/hotspot-main
>     > >     http://hg.openjdk.java.net/hsx/hotspot-comp
>     > >     http://hg.openjdk.java.net/hsx/hotspot-emb
>     > >     http://hg.openjdk.java.net/hsx/hotspot-gc
>     > >     http://hg.openjdk.java.net/hsx/hotspot-rt
>     > >
>     > > These are currently targeting jdk8, and will switch to jdk9 in the near
>     > > future.  At that point a new jdk8 stabilization repo would be created:
>     > >
>     > >     http://hg.openjdk.java.net/hsx/jdk8
>     > >
>     > > to be used instead of the existing hsx/hsx25 repo (hsx/hsx25 is
>     > > currently used only by the hotspot gatekeeper, Alejandro Murillo).
>     > >
>     > > Despite the length of this message, I think the naming change is
>     > > straightforward and will (slightly) simplify HotSpot development.
>     > > Since there are already HotSpot changes pending for jdk7u60, I want to
>     > > create the new repo within the next few days.  If you have questions
>     > > or feedback, please follow-up on the list.
>     > >
>     > > Background on the separate HotSpot version number:
>     > >
>     > > Because the same HotSpot source has been delivered simultaneously into
>     > > multiple JDK releases (e.g., builds of hs23 were delivered into jdk8
>     and
>     > > into jdk7u4, builds of hs22 were delivered into early jdk8 builds and
>     > > into jdk7u2, and so on), a separate HotSpot version facilitated
>     tracking
>     > > the sources as they propagated to different releases.  But at the same
>     > > time, it also imposed the overhead of having to map from a HotSpot
>     > > version to a JDK version and back again.  This is not always simple,
>     > > particularly for those that do not work with HotSpot on a day-to-day
>     > > basis.
>     > >
>     > > More recent hsx versions (hs24 and hs25), have really targeted only a
>     > > single JDK release (although a few builds of hs24 did go into both jdk8
>     > > and into jdk7u<N>).  We expect this trend to continue, and thus the
>     > > overhead of mapping from a HotSpot version to a JDK version is
>     > > unnecessary.
>     > >
>     > > -John