Backporting features [was Re: RFR (11u, XXL): Upstream/backport Shenandoah to JDK11u]

Fri Feb 7 16:19:53 UTC 2020

I think this is a discussion about the core of what an OpenJDK XXu update
project is intended to do and produce, and what it's primary directives are.

Let explain my opinion on the subject. Others' opinion will likely differ.

As I see it, the overriding purpose of an OpenJDK XXu project is to stabilize
and maintain the associated Java SE release implementation as it is, with no
intent to add any improvements or features. It is about applying bug fixes and
security updates to a given OpenJDK Java version, in order to keep it's current
use rock solid. It is not about "making it even better:". The place to make things
"even better" is in upstream versions. We have one of those leaving the station
every 6 months, with an LTS every three years, and those provide plenty of
opportunities to bring better-ness to the community.

While it may be tempting to try to improve things under the hood (e.g.
by making stuff faster, smaller, or better on some metric) or in visible ways
(e.g. by adding options, features, etc.) in stable or in just-now-released
versions, doing so will inherently risk the stability of the existing version,
and put the main purpose of the updates project at risk. The cut off
date for adding new things is (IMO) the XX.0.0 version. After that the only
things we bring in should be bug fixes and security updates, unless a
VERY high bar can be crossed. That bar is, IMO, "something is missing" in
the release.

The only two examples that I have seen rise above that bar so far are:

1. TLS 1.3 work: with the Java SE 8 spec updated via a maintenance release
with the specific intent of enabling support for TLS1.3 and ALPN in Java SE 8,
and with Oracle making it clear that they intend to update Oracle Java 8 to
include such support, I think that updating OpenJDK 8u accordingly with new
features is appropriate. It is a risky thing, but not one we have a real option of
avoiding. If we do not update OpenJDK 8u, it will be missing TLS 1.3 support.

2. JFR: As Oracle Java 8 JDKs have included JFR capability from the very
start of JDK 8. This capability was available to Java 8 developers in the
free Oracle JDK downloads from March 2014 until April 2019, and many users
have grown fond of using it in the field. The lack of JFR in OpenJDK 8
was "a missing feature" and was something people "lost" when moving from
Oracle JDK 8 to OpenJDK 8. The inclusion of JFR sources in upstream
OpenJDK 11 presented us with the opportunity to finally include that missing
feature to OpenJDK 8u.

For things that do not rise above the ""something is missing" bar in the main
updates project, curation choices about back-porting or adding features and
improvements to older versions can (and should) still be independently made
in downstream distributions and projects (e.g. Zulu, Corretto , DragonWell
RedHat's OpenJDK builds, Liberica, AOJ, and even Zing are all examples
of downstream distros that make such choices every day, and so do e.g.
shenandoah/jdk11). A choice to do that comes with the burden of maintainine
the difference over time and on an ongoing bases, and accepting the risk of
destabilization and additional complexity of backports that this may add. It
is a choice made independently by the group maintaining the specific
downstream distribution. It does not (and should not) affect the common
OpenJDK XXu project, or other downstream distributions that do not wish to
take on the same risks and burdens.

If we had a "HotSpot express" development model upstream (or even in or
across updates projects), I would view this quite differently. That model (which
was attempted for a short sliver of time in OpenJDK [1] but to my disappointment
did not stick) was meant to address the exact need here, and would have allowed
us to bring new HotSpot features (that do not affect the Java SE spec or version)
to older Java SE releases on a regular basis, such that e.g. improvements in
JITs, GCs, runtime features, platform support, and [jvm] bugfixes would flow to
all actively maintained versions. But we are not running with such a model, and
the current one clearly calls for code to be fixed but not improved once an
OpenJDK version has been released.

In this specific case, I see a clear and default path within OpenJDK: Focus on
stabilizing Shenandoah 2.0 in 15u & 16u to the point where it is no longer
an experimental feature. Getting it into 17u LTS (a mere 19 months from now)
will be a much more important and useful achievement then backporting it
into 11u in an experimental state.

[1] http://mail.openjdk.java.net/pipermail/jdk7u-dev/2011-August/000160.html

— Gil.

> On Feb 7, 2020, at 4:28 AM, Roman Kennke <rkennke at redhat.com> wrote:
> 
> Hi Gil,
> 
> Find my replies to your questions inline, below:
> 
>> As this is proposing a pretty significant backport of an experimental
>> feature
>> that is in-development in upstream versions, can you comment on how you
>> expect it to progress and evolve after initial integration into the stable,
>> in-maintenance-mode 11u?
>> 
>> E.g. here are some specifics:
>> 
>> - Do you expect that the 11u implementation will start diverging from
>> the upstream
>> path and start following a separate path towards a non-experimental
>> form? Or will it
>> closely match the upstream progress and changes, moving to non-experimental
>> form only when an upstream version includes a non-experimental version?
> 
> We intend to keep our current development practice around Shenandoah:
> keep the Shenandoah codebase synced across the 8u, 11u, and latest dev,
> as much as reasonably possible. Our experience with doing this for years
> already shows that the vast majority of the changes touch Shenandoah
> code only, and do not require changing any shared code. In the cases it
> did require changes in shared code, we erred on the side of caution, and
> either reworked (diverged from Shenandoah dev) to limit 8u/11u exposure,
> or made sure the changes are protected by UseShenandoahGC. That is a
> hard rule for our downstream repositories, and we intend to follow it in
> upstream as well.
> 
> Since codebases are more or less in sync, we think it would become
> non-experimental pretty much when upstream becomes non-experimental.
> 
>> - Do you expect to advocate for porting of bug fixes only, or also of
>> features and
>> improvements from upstream versions? [In my terminology, a bug fix is
>> something
>> like "no longer crashes when X happens", an improvement is something
>> like  "runs
>> twice as fast" or "has generational collection", and a feature is
>> something like e.g.
>> "can be configured with new flag X to proactively adjust heap size over
>> time".
>> E.g. if the redesign from Shenandoah 1.0 to Shenandoah 2.0 had happened
>> after
>> this initial backport was integrated, would you later be advocating for
>> a backport of
>> the redesign to 11u?
> 
> Until now, we backported everything, including new features, as long as
> we can do so without touching shared code. But there are large items
> that require lots of fiddling with shared code. Two prominent examples
> are: a) concurrent class unloading in jdk14, which requires a lot of VM
> infrastructure to support; b) loop strip mining in jdk11, which required
> significant work in C2. Both are deemed too risky to backport to
> previous releases.
> 
> The LRB rework was special. While it required a massive amount of shared
> code changes (in our shenandoah/jdk11 tree), most of them reduced
> overall shared code exposure. The patch we have presented for
> integration to 11u already includes it, and this is indeed why the
> shared code exposure is quite low.
> 
> There is a technical advantage of backporting most of the stuff: the
> backporting process is simplified. The vast majority of Shenandoah dev
> -> 11u backports applies cleanly and requires minimum review and retesting.
> 
>> - If it is more than just bug fixes that you propose tracking, then for
>> how long? e.g.
>> once 17u comes out in Sep. 2021 (19 months from now), will you advocate that
>> new Shenandoah improvements or features being worked on in OpenJDK 18u
>> be backported top 11u? Or will there be some sunset where we say "11 has
>> what
>> 11 has, if you want new improvements and features, move to new version X"?
> 
> See above, it is judged on ad-hoc basis. If it can be done without
> touching shared code, then we'd probably backport it to 11u after
> reasonable baking time in upstream.
> 
>> - Would you advocate for backporting from upstream versions to all
>> intermediate
>> versions? e.g. from a 16EA  to 15.0.2 in Jan 2021, or to an 13.0.6 if
>> one was still
>> being built?, or just to 11u?
> 
> We have not made a firm decision regarding intermediate versions. Our
> soft policy is to backport critical issues to STS releases, so that
> users would get them earlier than next STS releases. Then we basically
> relied on 6-month switchover to the new STS. If any STS continues to
> deliver releases after next STS is available, we can indeed consider
> extending that soft policy to "previous" STS releases. But the priority
> for that work would be rather low.
> 
>> - Do you expect to start backporting other upstream features or
>> redesigns aimed at
>> supporting the maturation of Shenandoah in 11u, but that would not be
>> needed in 11u
>> for other reasons? The example of
>> https://bugs.openjdk.java.net/browse/JDK-8230565
>> comes to mind, as I expect the new LRB might be similarly susceptible to C2
>> scheduling load barriers across safepoints, and may need a similar
>> redesign of
>> the C2 load barrier in order to stabilize an LRB-based Shenandoah
>> implementation.
>> Would you advocate for such a redesign to be backported into the stable
>> 11u C2
>> as well? Would it make sense to introduce the Shenandoah backport if we
>> do not
>> intend to later follow through with such supporting (but wider impacting
>> on 11u) items?
> 
> Since the primary mandate for LTS releases is to avoid destabilizing
> changes, any problem with it would have to be band-aided in one way or
> the other. It is not very different from discovering the bugs after the
> initial integration to 11u: since that is LTS version, you do what you
> can to make it work. Re: JDK-8230565, we do not intend to either use or
> backport it, as our C2 barriers work differently.
> 
>> - Do you expect that the initial backport, or it's later evolution, will
>> require changes
>> to critical shared code components in 11u that would not be needed
>> otherwise in
>> 11u, and are not yet included in e.g. any upstream LTS version? E.g.
>> your suggetions
>> for https://bugs.openjdk.java.net/browse/JDK-8209686 seems like a
>> reasonable and
>> not-too-big change, but one that 11u has no need for and would likely
>> not bring in to
>> C2 for any other reason. How to we choose if and when to add such things
>> to the
>> stable, in-maintenance-mode 11u (and risk destabilizing its core
>> functionality
>> inadvertently)?
> 
> JDK-8209686 was a preparatory change. It was pretty much contained in
> the big Shenandoah patch, and I thought it makes sense to review and
> push that part separately. It follows the established practice in JDK
> Updates Project: if we can, we backport simple stuff that makes
> subsequent backports much cleaner.
> 
> Best regards,
> Roman
> 
>> — Gil.
>> 
>>> On Feb 6, 2020, at 11:43 AM, Roman Kennke <rkennke at redhat.com
>>> <mailto:rkennke at redhat.com>> wrote:
>>> 
>>> Hello all,
>>> 
>>> This is an update on the proposed patch.
>>> 
>>> - It is rebased on latest jdk11u-dev, and in particular on the
>>> backport of:
>>> https://bugs.openjdk.java.net/browse/JDK-8209686
>>> 
>>> ... which considerably improves C2 shared-code-changes.
>>> 
>>> - Includes the latest bunch of Shenandoah/jdk11u backports by Aleksey:
>>> https://mail.openjdk.java.net/pipermail/shenandoah-dev/2020-February/011417.html
>>> 
>>> Full webrev:
>>> http://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.03-all/
>>> 
>>> Shared-code webrev:
>>> http://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.03-shared/
>>> 
>>> Please review/test that instead of the previous patches.
>>> 
>>> Thanks,
>>> Roman
>>> 
>>>> Hello,
>>>> 
>>>> The Shenandoah GC has been integrated into jdk12 a little over a year
>>>> ago:
>>>> 
>>>> http://hg.openjdk.java.net/jdk/jdk/rev/9c18c9d839d3
>>>> 
>>>> The Shenandoah team has been maintaining the Shenandoah-enabled jdk11u
>>>> downstream repository since the inception of jdk11 under:
>>>> 
>>>> http://hg.openjdk.java.net/shenandoah/jdk11
>>>> 
>>>> In order to make it more useful and accessible for users, we would like
>>>> to make it available in upstream jdk11u. The Shenandoah team at Red Hat
>>>> intends to maintain it the same way it had been maintained in
>>>> shenandoah/jdk11, in the course of regular jdk11u maintenance Red Hat
>>>> already does.
>>>> 
>>>> Thanks to recent changes in Shenandoah over the last year, we are now
>>>> able to fit the GC interface in jdk11 almost exactly. There are a few
>>>> exceptions though. We tried to follow the following pattern for all
>>>> shared-code changes where it made sense:
>>>> 
>>>> #if INCLUDE_SHENANDOAH_GC
>>>>  if (UseShenandoahGC) {
>>>>    ...
>>>>  }
>>>> #endif
>>>> 
>>>> This ensures that the Shenandoah-specific changes are cut out of the
>>>> build when building without Shenandoah, and avoid those code paths at
>>>> runtime when running without Shenandoah.
>>>> 
>>>> Also, there are a few places where this was not possible or where it
>>>> didn't seem feasible, but we tried to keep them few and obvious.
>>>> Whenever this is the case, we are mirroring as close as possible what we
>>>> have in jdk/jdk.
>>>> 
>>>> Architecture-wise, Shenandoah runs on x86_64, aarch64, x86_32. Other
>>>> architectures still build fine, and Shenandoah gets automatically
>>>> disabled during configure if not supported. OS-wise, Shenandoah runs on
>>>> Linux, Windows, and is known to run on MacOS X and Solaris too.
>>>> 
>>>> I wasn't sure which form this should take. I decided to start with the
>>>> easiest possible form this could take: a simple patch/webrev and an RFR.
>>>> Let me know if you need a JIRA issue or any other formalities to
>>>> track that.
>>>> 
>>>> Full webrev:
>>>> https://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.02-all/
>>>> 
>>>> Webrev only for shared-code changes:
>>>> https://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.02-shared/
>>>> 
>>>> The webrevs are based on and depend on the arraycopy patch proposed for
>>>> review here:
>>>> 
>>>> https://mail.openjdk.java.net/pipermail/jdk-updates-dev/2020-January/002396.html
>>>> 
>>>> Testing:
>>>> The code has been tested many many times, continuously while maintaining
>>>> it. It has also served as the basis for RPMs and shipped to customers.
>>>> We are running all sorts of tests of the hotspot/jtreg testsuite, in
>>>> particular hotspot_gc_shenandoah, CTW tests, the regular tier* tests,
>>>> several popular benchmarks (specjvm, specjbb on a regular basis), on a
>>>> nightly basis.
>>>> 
>>>> Please let me know what you think.
>>>> 
>>>> Thanks,
>>>> Roman
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> 
> 