RFR (11u, XXL): Upstream/backport Shenandoah to JDK11u

[Gil Tene]

Roman,

As this is proposing a pretty significant backport of an experimental feature
that is in-development in upstream versions, can you comment on how you
expect it to progress and evolve after initial integration into the stable,
in-maintenance-mode 11u?

E.g. here are some specifics:

- Do you expect that the 11u implementation will start diverging from the upstream
path and start following a separate path towards a non-experimental form? Or will it
closely match the upstream progress and changes, moving to non-experimental
form only when an upstream version includes a non-experimental version?

- Do you expect to advocate for porting of bug fixes only, or also of features and
improvements from upstream versions? [In my terminology, a bug fix is something
like "no longer crashes when X happens", an improvement is something like  "runs
twice as fast" or "has generational collection", and a feature is something like e.g.
"can be configured with new flag X to proactively adjust heap size over time".
E.g. if the redesign from Shenandoah 1.0 to Shenandoah 2.0 had happened after
this initial backport was integrated, would you later be advocating for a backport of
the redesign to 11u?

- If it is more than just bug fixes that you propose tracking, then for how long? e.g.
once 17u comes out in Sep. 2021 (19 months from now), will you advocate that
new Shenandoah improvements or features being worked on in OpenJDK 18u
be backported top 11u? Or will there be some sunset where we say "11 has what
11 has, if you want new improvements and features, move to new version X"?

- Would you advocate for backporting from upstream versions to all intermediate
versions? e.g. from a 16EA  to 15.0.2 in Jan 2021, or to an 13.0.6 if one was still
being built?, or just to 11u?

- Do you expect to start backporting other upstream features or redesigns aimed at
supporting the maturation of Shenandoah in 11u, but that would not be needed in 11u
for other reasons? The example of https://bugs.openjdk.java.net/browse/JDK-8230565
comes to mind, as I expect the new LRB might be similarly susceptible to C2
scheduling load barriers across safepoints, and may need a similar redesign of
the C2 load barrier in order to stabilize an LRB-based Shenandoah implementation.
Would you advocate for such a redesign to be backported into the stable 11u C2
as well? Would it make sense to introduce the Shenandoah backport if we do not
intend to later follow through with such supporting (but wider impacting on 11u) items?

- Do you expect that the initial backport, or it's later evolution, will require changes
to critical shared code components in 11u that would not be needed otherwise in
11u, and are not yet included in e.g. any upstream LTS version? E.g. your suggetions
for https://bugs.openjdk.java.net/browse/JDK-8209686 seems like a reasonable and
not-too-big change, but one that 11u has no need for and would likely not bring in to
C2 for any other reason. How to we choose if and when to add such things to the
stable, in-maintenance-mode 11u (and risk destabilizing its core functionality
inadvertently)?

— Gil.

> On Feb 6, 2020, at 11:43 AM, Roman Kennke <rkennke at redhat.com> wrote:
> 
> Hello all,
> 
> This is an update on the proposed patch.
> 
> - It is rebased on latest jdk11u-dev, and in particular on the backport of:
> https://bugs.openjdk.java.net/browse/JDK-8209686
> 
> ... which considerably improves C2 shared-code-changes.
> 
> - Includes the latest bunch of Shenandoah/jdk11u backports by Aleksey:
> https://mail.openjdk.java.net/pipermail/shenandoah-dev/2020-February/011417.html
> 
> Full webrev:
> http://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.03-all/
> 
> Shared-code webrev:
> http://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.03-shared/
> 
> Please review/test that instead of the previous patches.
> 
> Thanks,
> Roman
> 
>> Hello,
>> 
>> The Shenandoah GC has been integrated into jdk12 a little over a year ago:
>> 
>> http://hg.openjdk.java.net/jdk/jdk/rev/9c18c9d839d3
>> 
>> The Shenandoah team has been maintaining the Shenandoah-enabled jdk11u
>> downstream repository since the inception of jdk11 under:
>> 
>> http://hg.openjdk.java.net/shenandoah/jdk11
>> 
>> In order to make it more useful and accessible for users, we would like
>> to make it available in upstream jdk11u. The Shenandoah team at Red Hat
>> intends to maintain it the same way it had been maintained in
>> shenandoah/jdk11, in the course of regular jdk11u maintenance Red Hat
>> already does.
>> 
>> Thanks to recent changes in Shenandoah over the last year, we are now
>> able to fit the GC interface in jdk11 almost exactly. There are a few
>> exceptions though. We tried to follow the following pattern for all
>> shared-code changes where it made sense:
>> 
>> #if INCLUDE_SHENANDOAH_GC
>>  if (UseShenandoahGC) {
>>    ...
>>  }
>> #endif
>> 
>> This ensures that the Shenandoah-specific changes are cut out of the
>> build when building without Shenandoah, and avoid those code paths at
>> runtime when running without Shenandoah.
>> 
>> Also, there are a few places where this was not possible or where it
>> didn't seem feasible, but we tried to keep them few and obvious.
>> Whenever this is the case, we are mirroring as close as possible what we
>> have in jdk/jdk.
>> 
>> Architecture-wise, Shenandoah runs on x86_64, aarch64, x86_32. Other
>> architectures still build fine, and Shenandoah gets automatically
>> disabled during configure if not supported. OS-wise, Shenandoah runs on
>> Linux, Windows, and is known to run on MacOS X and Solaris too.
>> 
>> I wasn't sure which form this should take. I decided to start with the
>> easiest possible form this could take: a simple patch/webrev and an RFR.
>> Let me know if you need a JIRA issue or any other formalities to track that.
>> 
>> Full webrev:
>> https://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.02-all/
>> 
>> Webrev only for shared-code changes:
>> https://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.02-shared/
>> 
>> The webrevs are based on and depend on the arraycopy patch proposed for
>> review here:
>> 
>> https://mail.openjdk.java.net/pipermail/jdk-updates-dev/2020-January/002396.html
>> 
>> Testing:
>> The code has been tested many many times, continuously while maintaining
>> it. It has also served as the basis for RPMs and shipped to customers.
>> We are running all sorts of tests of the hotspot/jtreg testsuite, in
>> particular hotspot_gc_shenandoah, CTW tests, the regular tier* tests,
>> several popular benchmarks (specjvm, specjbb on a regular basis), on a
>> nightly basis.
>> 
>> Please let me know what you think.
>> 
>> Thanks,
>> Roman
>> 
>> 
>> 
>> 
>> 
>> 
>>