Backporting features [was Re: RFR (11u, XXL): Upstream/backport Shenandoah to JDK11u]

[Andrew John Hughes]

On 10/02/2020 16:30, Gil Tene wrote:
> 

snip...

>>
>>> For things that do not rise above the ""something is missing" bar in
>>> the main updates project, curation choices about back-porting or
>>> adding features and improvements to older versions can (and should)
>>> still be independently made in downstream distributions and projects
>>> (e.g. Zulu, Corretto , DragonWell RedHat's OpenJDK builds, Liberica,
>>> AOJ, and even Zing are all examples of downstream distros that make
>>> such choices every day, and so do e.g.  shenandoah/jdk11).
>>
>> Certainly, this happens. However, we are at some risk of Balkanization
>> of JDK 11 distributions, with downstream vendors independently
>> packaging extensions. This is a Bad Thing and a waste of effort. If we
>> can share resources without risking our users' stability then everyone
>> benefits.
>>

To me, this is the key point here and a far greater risk than the
theoretical one of a backported feature causing instability.

Having Shenandoah available in 11u is not a hypothetical. It is
something that exists today and is being shipped to end users. This has
now been the case for a number of years.

For every such downstream, that is development and testing work that is
being expended on a fork to the detriment of upstream OpenJDK. The
stability (or lack thereof) of upstream OpenJDK becomes completely
irrelevant if no-one is using it and is instead reliant on some
downstream that works for them.

In the case of Shenandoah specifically, our testing for the final
release of 11u is currently split between builds based on the Shenandoah
OpenJDK 11 backport and vanilla OpenJDK 11. To my mind, it would clearly
be much preferable to be working solely on OpenJDK 11 itself, and
healthier for this project.

This is nothing new. OpenJDK 6 & 7 were both largely unused in their
vanilla versions, with various GNU/Linux distributions downstream using
various forks. This largely stemmed from the nascent nature of OpenJDK
at the time, and there has been active effort since to move away from
this model towards inclusion in OpenJDK projects, and eventually
mainline, since.

I can understand that the fear here comes from a feature being
backported that is of no use to some, but then causes problems. OpenJDK
is a big enough project that there will be elements like that for all of
us. In my experience, the best way to resolve such fears is to have as
many people as possible working and testing the *same* codebase. There
are few things more annoying than discovering a bug, only to find
someone already found and fixed that issue a long time ago in their
downstream.

I do find it strange that the TLS 1.3 and JFR backports to 8u are being
considered to be exemptions here. Both of these are still very much a
work-in-progress, when compared with Shenandoah. Personally, I am
currently more concerned about both these backports causing disruption
in 8u than Shenandoah in 11u, a backport of which has been available in
a public repository for some time and has had widespread testing. In
comparison, the JFR backport was, until recently, unable to be built
using the standard bootstrap process with JDK n-1, and the TLS 1.3
backport is still in the planning stages.

As to an "HotSpot Express" model, this has similar concerns to the
existing problem of downstreams. In IcedTea for 8u, at present, we allow
a choice of three different HotSpots (8u + AArch64 local to IcedTea, the
aarch64/shenandoah-jdk8u HotSpot & the aarch32-port HotSpot) in the
manner you seem to be suggesting. Doing so means triple the testing for
every new build integration. Coupling the forked HotSpot to upstream
OpenJDK's build system in some way doesn't remove the need to test the
alternate configuration.

Unless you intend to somehow start this process upstream, and make it
again possible for a e.g. OpenJDK 15 HotSpot to be simply slotted in and
used with 11u, I don't see how it reduces any of the additional work
needed to maintain multiple forks. Even a HSX in OpenJDK 15 would
require continuous maintenance, because I doubt Oracle want to maintain
that.

In short, while I appreciate the concerns, I think upstream integration
is the best solution available, short of convincing the people who want
these features not to use 11u at all and switch to the latest OpenJDK.

Thanks,
-- 
Andrew :)

Senior Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net)
Fingerprint = 5132 579D D154 0ED2 3E04  C5A0 CFDA 0F9B 3596 4222
https://keybase.io/gnu_andrew