Backporting features [was Re: RFR (11u, XXL): Upstream/backport Shenandoah to JDK11u]

[Gil Tene]

> On Feb 7, 2020, at 10:13 AM, Andrew Haley <aph at redhat.com> wrote:
> 
> First, let me state that in this discussion I am doing my best to
> ensure that I am not biased by my job at Red Hat: despite my signature
> below, I am writing as the JDK 11 Updates maintainer. I will try to
> remain neutral and act in the best interests of the project, as I hope
> will everyone else in this discussion.

Agreed. While we all wear multiple hats, I would hope and assume that
the discussions here relating to OpenJDK updates are driven by our
opinions on what is best for the community that consumes those updates.
I know mine is.

> 
> On 2/7/20 4:19 PM, Gil Tene wrote:
>> I think this is a discussion about the core of what an OpenJDK XXu update
>> project is intended to do and produce, and what it's primary directives are.
>> 
>> Let explain my opinion on the subject. Others' opinion will likely differ.
> 
> Probably.
> 
>> As I see it, the overriding purpose of an OpenJDK XXu project is to
>> stabilize and maintain the associated Java SE release implementation
>> as it is, with no intent to add any improvements or features. It is
>> about applying bug fixes and security updates to a given OpenJDK
>> Java version, in order to keep it's current use rock solid. It is
>> not about "making it even better:". The place to make things "even
>> better" is in upstream versions. We have one of those leaving the
>> station every 6 months, with an LTS every three years, and those
>> provide plenty of opportunities to bring better-ness to the
>> community.
> 
> Well, yes, kinda sorta: we must not be foolhardy, because the safety
> of our users and the reputation of OpenJDK is at stake.
> 
> However, OpenJDK is free software, so another issue at stake is
> freedom: we want people to have a choice. Therefore, we should ensure,
> as much as we can, that our users are not forced into the arms of
> proprietary software vendors because of features lacking in equivalent
> OpenJDK releases.

Choice is good. But we are talking about two conflicting things: stability
and evolution. And (as an opinion) I don't think that people need to be
forced to choose  evolution over stability, especially when alternatives
for evolution are more readily available in OpenJDK than ever before
(a new version comes out every 6 months).
I would guess that for every OpenJDK user that wants to see 11u or 8u
have new features (and is not willing to go to 13u or 14u for those), there
are at least 10 that simply want 11u and 8u to remain rock-solid, and only
take on changes that must be taken to improve stability and security.

The historical cadence of Java and OpenJDK releases has been simple:

- Versions (7, 8, 9, 10, 11, 12, 13…) bring new features and improvements

- Within a version, updates bring bug fixes and add security updates, and
  make sure the current version is updated to run well on the latest/greatest
  versions of OSs.
  [** We have certainly had exceptions to the above, and I had already
   listed two examples of features that IMO closed a gap large enough
   to be clearly above the bar, but the bar is very high]

I believe that the vast majority of Java users expect the same cadence
and behavio to remainr, with stability increasing over time within a version,
and capabilities improving over time across versions. It has been this way
for decades. and it drives how and when people choose their jdk
versions.

If we are going to change that, and start offering a choice between
evolution and stability within an OpenJDK versions, it should be a choice,
and not a mandate that applies to all consumers of e.g. the 11u project.
Such a choice would require two separate things: an 11u, and an 11uPlus.
If some folks want to start and run with an OpenJDK 11u Plus project thing
that goes far beyond maintaining OpenJDK 11u, and seeks to improve it
with new features over time, I'm all for that. I may even contribute to it.
But please please please don't do this as onl;y thing available in the
11u and 8u projects.

If we do want to venture down the "let's bring features back to 11u
(and maybe 8u)" path, I think that the hotspot-express model was the
absolute right way to do this, and that we should consider starting a
hotspot-express project for that purpose. The project would not be
OpenJDK-version specific, but would instead build hotspot JVMs that
are able to run any Java version (across a wide spectrum, e.g. starting
with 11u or 8u). One would then be able to bundle a hotspot express
JVM with an 11u, 13u, or 17u OpenJDK build (replacing the hotspot
component within) and get the various JVM-only improvements, including
new GCs, JIT improvements, runtime features, etc. (but no language
features or class library/api changes)

We could even change e.g. the 11u project build system allow one to
choose between packaging the built-in hotspot (the one we primarily
update for bug fixes and security reasons) or the hotspot-express jvm.

And who knows: over time, if hotspot-express does well enough, and
establishes a track record of evolution and stability at the same time,
we may be able to drop the built-in hotspot altogether. But I would
anticipate that to take several years top establish.

> 
>> While it may be tempting to try to improve things under the hood
>> (e.g.  by making stuff faster, smaller, or better on some metric) or
>> in visible ways (e.g. by adding options, features, etc.) in stable
>> or in just-now-released versions, doing so will inherently risk the
>> stability of the existing version, and put the main purpose of the
>> updates project at risk.
> 
> I understand. However, if we can determine that a feature backport is
> 
>    a. Desired by a number of downstream distributors (and their
>    users), and
> 
>    b. Sufficiently guarded by flags (e.g. #ifdef) that we can
>    determine statically that a feature has no effect if not
>    enabled/configured
> 
> ... I believe we will not be taking undue risks. Downstream
> distributors can take the decision to enable features or not.
> 
> You might argue that even when not configured there is some slight
> risk, but surely a risk has to be plausible, not just theoretically
> possible.
> 
>> For things that do not rise above the ""something is missing" bar in
>> the main updates project, curation choices about back-porting or
>> adding features and improvements to older versions can (and should)
>> still be independently made in downstream distributions and projects
>> (e.g. Zulu, Corretto , DragonWell RedHat's OpenJDK builds, Liberica,
>> AOJ, and even Zing are all examples of downstream distros that make
>> such choices every day, and so do e.g.  shenandoah/jdk11).
> 
> Certainly, this happens. However, we are at some risk of Balkanization
> of JDK 11 distributions, with downstream vendors independently
> packaging extensions. This is a Bad Thing and a waste of effort. If we
> can share resources without risking our users' stability then everyone
> benefits.
> 
>> In this specific case, I see a clear and default path within
>> OpenJDK: Focus on stabilizing Shenandoah 2.0 in 15u & 16u to the
>> point where it is no longer an experimental feature. Getting it into
>> 17u LTS (a mere 19 months from now) will be a much more important
>> and useful achievement then backporting it into 11u in an
>> experimental state.
> 
> Here, I think you may have a point: there is a question about whether
> experimental features should be back-ported. However, there are
> already experimental features in JDK 11, so it's not as if they're
> completely verboten in LTS releases. We should decide on the actual
> demand (the benefit) versus the actual intrusiveness (the cost) rather
> than in-principle arguments which may have more to do with theology
> than engineering.

I'd like to think that I'm a pragmatist, and that this is not a theological
discussion in either direction.

IMO the updates to an already released version of OpenJDK is not a
place to introduce new experiments… We have plenty of places to
evolve and mature experimental features, including side projects, as
well as upstream versions. The only experimental things I know of in
JDK11 (or later) releases were put there at version .0.0. AFAIK (I may
be wrong), since Java 7, there have not been any experimental features
added to a JDK after its initial .0 GA release.

But my worries apply not only to experimental things. They also apply to
fully baked ones as well. I believe that the bar for adding any new feature
to a stable JDK updates project needs to be very high. It is not just a
matter of balance between risk and benefit. It is a matter if the need being
significant enough to warrant any risk at all.

E.g. the non-experimental Dynamic CDS Archives feature is really cool.
But I would not want to see it backported into an OpenJDK 11.0.8.

E.g. when a fully baked, value-types supporting C2 change eventually
mature upstream and is no longer experimental I will probably be
strongly against back-porting those potentially destabilizing C2 changes
to 11u.

A hotspot-express model would allow us to box this, and provide a choice
for users, between a "latest-greatest" jvm with new features, and a "same
jvm that was initially shipped with the version" with stability and security
improvements. But without such a choice, I think the default is obvious.

> 
> --
> Andrew Haley  (he/him)
> Java Platform Lead Engineer
> Red Hat UK Ltd. <https://www.redhat.com>
> https://keybase.io/andrewhaley
> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671