RFR (11u, XXL): Upstream/backport Shenandoah to JDK11u

Gil Tene gil at azul.com
Wed Jul 8 21:39:52 UTC 2020 


> On Jul 8, 2020, at 11:31 AM, Roman Kennke <rkennke at redhat.com> wrote:
> 
> Hi Gil & all,
> 
> I put in some extra work and had a deep look at the remaining unguarded
> changes in shared-code, with an eye to either eliminating them
> altogether, or - where this is not possible - guard them with build-
> time #if INCLUDE_SHENANDOAHGC or similar. The guiding principle in this
> effort has been that building without Shenandoah (--with-jvm-features=-
> shenandoahgc) would compile *the exact same* code that it does now. I
> believe I achieved this with the following proposed changes:
> 
> Full webrev (incl. Shenandoah-only files):
> https://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.06-all/
> 
> Shared-code-only changes:
> https://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.06-shared/
> 
> The latter webrev exludes all files that are only compiled with
> Shenandoah enabled.
> 
> Also, compared to previous webrevs, this excludes Shenandoah by
> default.
> 
> As far as I can tell, this literally compiles the same code into
> libjvm.so as it does now, when building without Shenandoah. The only
> exception is the inclusion of the global flag UseShenandoahGC, which is
> by-design. When this is enabled in a build without Shenandoah, it
> prints the appropriate error message and exist (exactly like with any
> other GC that is not built-in).

For the purpose of this thought exercise, I think we could consider that flag
addition as a separate change (which would happen before all the other
stuff). So that we can consider the "would compile *the exact same* code"
statements in the context of a builds that already include the flag.

> 
> I've been thinking about if we can come up with a mechanical proof of
> correctness. The first best thing would have been reproducible builds,
> but we don't have that (and the UseShenandoahGC flag would mess it up
> too). The other option would be to dump preprocessed sources and diff
> them, but that is disturbed by the necessary inclusion of
> "utilities/macros.hpp" which gives us the INCLUDE_SHENANDOAHGC macro.

I think that a mechanized way to verify the assumption will be key, as you
will want it verified not just initially, but with every future change. figuring
out how to do that is a key question...

Yes, reproducible builds would be great, but we probably don't want to
hold our breath for that. Maybe something around comparing individual
object file contents that go into the final link can work?

> 
> Which leaves us with carefully inspecting the webrev by multiple sets
> of eyes and brains.

Eye and brain inspection is unlikely to catch innocent looking "leaks" in
this model 6 months after the initial merge effort is completed, when
someone touches a line on the wrong side of an ifdef, or makes a
change to the scope of an ifdef or to what turns it on it.

I think that an ongoing means of verification is needed as part of a
solution like this for it to be viable maintainable. I'm not sure what
that would look like. But it may be worth thinking up some techniques.

I'm also not sure how one would police future code changes and
backports of fixes from usptream versions, unless each and every
review of code carefully takes into consideration the "is this Shenandoah
related or not? Should it go in ifdefs or not?" questions.

Can we come up with systemic ways to know that a code change being
backported is Shenandoah-related? E.g. if the upstream code shape and
ifdef'ing rules remained similar to the downstream, we may be able to
keep things workable, but I suspect trhat the upstream (especially after
15u) will be tempted to remove the strict ifdef'ing pretty quickly, and start
placing code changes in common code, and as those changes will likely
be needed in backporting gixes to the "Shenandoah enabled 11u" mode,
carefully examination on a case by case basis will be needed, at least as
long as "no shenandoah exists" downstreams of 11u still depend on it
for updates.

> 
> Gil, would the change like this ease your concerns?

If we could provably show that no code differences exist in builds that
do no choose to include the change, it would be hard for me to argue
against upstream inclusion based on it having a potential for breaking
a downstream distro's stability, no matte rhow neccesary or not it is…

So yes, the proposed approach could work if we can figure out
the technical ways to achieve it and "keep it achieved" for some time
into the future.

We will likely need to be able to mechanically verify that things remain
like this even as code is introduced in the future. Manual review cannot
achieve that alone probably, so we will likely need some build tooling
to verify this every time.

How long will we need to be able to keep proving this for? I'm not sure.
One could argue that it is several years, but I can see an argument that
says that after enough time (2 years?) and enough production exposure
of the 11u Shenandoah upstream thing, the world will gain enough
confidence in it that we no longer need to keep proving the quality suggested.
The most telling sign will be whether or not downstream 11u distros that
do not include Shenandoah keep producing updates. Once none of
those non-Shenandoah build updates happen, there will obviously be
no point in keeping the thing proveable in every update…

I for one can imagine making Zulu 11 versions of both kinds, letting
people choose, and even encouraging new users to use the Shanadoah-enabled
one. Once the number of people who seek 11u updates that don't have
Shenandoah in them any more drops to some truly tiny number (tiny enough
that we feel we can force them to the other updates], I'd happily try to force the
issue with them and scratch Zulu and it's mass number of downstream users
from the "people who may care" list…

> Thank you,
> Roman
> 
> 
> On Fri, 2020-06-26 at 23:39 +0000, Gil Tene wrote:
>> As I noted  before, we have serious reservations about adding major
>> new features to a
>> stable LTS that is upstream of all of 11u. Based on daily
>> interactions with tons of OpenJDK
>> end users, I can say that the vast majority of end users that are
>> ramping up on 11u are
>> demanding stability and maturity above all else. The addition of such
>> major changes in
>> an 11u update will cause many of them to stall updates, and
>> potentially stall the uptake
>> of 11u altogether, waiting for 11u to actually stabilize first
>> (evidenced by no longer doing
>> such things), so they can keep up with fixes and security issues
>> without having to take on
>> potential regressions driven by late added or experimental features.
>> 
>> If you want a modern GA'ed OpenJDK with all the latest features in
>> it, use one. Please
>> please please don't force people who choose to stick with a given
>> OpenJDK version for
>> stability, and knowingly did not move to a later OpenJDK version, to
>> deploy the newly
>> implemented features that you like by pushing for back-porting them
>> to the upstream
>> project that they get their security updates from...
>> 
>> I can see a potentially good case for a downstream-from-11u project
>> that incorporates
>> ongoing development (as opposed to bug fixes) from later upstream
>> versions, and would
>> be happy to collaborate on one. But the 11u that is upstream of all
>> other 11u's should not
>> be seen as a developer's playground, or a place to add cool new
>> features to a stable
>> release. The upstream versions are there for that purpose. The valid
>> interests of
>> deep-bench, self-supporting groups making use of OpenJDK, who can
>> take on the stability
>> risks involved due to the depth of their technical bench, should not
>> overcome the interests
>> of the millions of consumers of the 11u builds that do not have that
>> luxury, and who
>> need and expect stable (in the changing only where necessary sense)
>> 11u releases.
>> 
>> I'd like to point out that 13u already includes Shenandoah. 15u
>> (expected in 3 months)
>> already includes Shenandoah as a non-experimental feature. And we can
>> all collaborate
>> on back-porting bug-fixes to those to updates to keep a lively
>> version of OpenJDK that
>> includes Shenandoah up to date from a bug fix and security update
>> point of view. For
>> those insisting on waiting for an LTS (on the assumption that LTSs
>> are long term stable
>> and don't get those sort of messing-with-after-the-fact done to them,
>> mind you), 17u will
>> be out in 15 months.
>> 
>> So we are not talking about some far-away future for end users that
>> want Shenandoah.
>> It is here, now, in multiple consumable forms, and available for
>> people not looking for
>> the stability of an LTS, and even for people who do (as they can use
>> the Red Hat 11u
>> builds). It it will be available in more forms very soon with no
>> effort, and we don't have
>> to destabilize 11u for others to make that happen.
>> 
>> — Gil.
>> 
>>> On Jun 26, 2020, at 1:51 PM, Mathiske, Bernd <mathiske at amazon.com>
>>> wrote:
>>> 
>>> Hi Roman,
>>> 
>>> We are also in favor of upstreaming Shenandoah to jdk11u, for the
>>> same reasons Aditya listed, and we agree with everything else he
>>> has written. Similar to Aditya’s message below, we believe that
>>> Shenandoah has strong demand and potential for those running
>>> jdk11u.  Targeting jdk11u is the most efficient way to reach a
>>> large user base and to enable production use of Shenandoah.
>>> 
>>> At Amazon, we plan to continue to contribute to Shenandoah, and we
>>> fully intend to help maintain Shenandoah in jdk11u.
>>> 
>>> Best,
>>> 
>>> Bernd, Kelvin, Azeem
>>> 
>>> On 6/26/20, 10:36 AM, "jdk-updates-dev on behalf of Aditya
>>> Mandaleeka" <jdk-updates-dev-retn at openjdk.java.net on behalf of
>>> adityam at microsoft.com> wrote:
>>> 
>>>   Hi Roman,
>>> 
>>>   I am in favor of this proposed upstreaming of Shenandoah into
>>> jdk11u. Microsoft
>>>   has some long-term commitments to Java 11 customers and having
>>> Shenandoah
>>>   included in jdk11u upstream would definitely be well-received by
>>> my team and by
>>>   our customers. There is demand for this kind of GC option on
>>> OpenJDK 11, and it
>>>   would be beneficial to the community for it to be available on
>>> the upstream
>>>   OpenJDK rather than only in specific downstream releases.
>>> Coupled with the fact
>>>   that Shenandoah has been tested on JDK 11 for quite a while now,
>>> I think this
>>>   upstreaming makes sense.
>>> 
>>>   You mentioned in your original mail that your team at RedHat
>>> intends to maintain
>>>   this proposed jdk11u version the same way it has been maintained
>>> in
>>>   shenandoah/jdk11. That is great--from what I’ve seen in these
>>> past few months
>>>   that I’ve been involved, I’ve been impressed by the way
>>> backports to
>>>   shenandoah/jdk11 are being handled, the effort that is made to
>>> keep the
>>>   Shenandoah code closely aligned between sh/11 and jdk/jdk, and
>>> the care taken to
>>>   ensure that the impact on shared code is kept to a minimum.
>>> 
>>>   At Microsoft, we’ve already been contributing patches and
>>> investigating and
>>>   reporting issues based on our testing of Shenandoah, both on
>>> jdk/jdk and
>>>   shenandoah/jdk11. We intend to continue this kind of
>>> involvement, so if this
>>>   upstreaming to jdk11u goes through, your team will also have our
>>> support in
>>>   maintaining Shenandoah there.
>>> 
>>>   Thanks,
>>>   Aditya
>>> 
>>>   -----Original Message-----
>>>   From: jdk-updates-dev <jdk-updates-dev-retn at openjdk.java.net> On
>>> Behalf Of Roman Kennke
>>>   Sent: Thursday, June 25, 2020 4:16 AM
>>>   To: jdk-updates-dev at openjdk.java.net
>>>   Subject: Re: RFR (11u, XXL): Upstream/backport Shenandoah to
>>> JDK11u
>>> 
>>>   I would like to revive this RFR, I believe we haven't come to a
>>>   conclusion yet. I still think it would be very useful to have
>>> the
>>>   Shenandoah GC included in jdk11u upstream. I have heard from
>>> several
>>>   vendors who would like to ship Shenandoah, but maintaining the
>>> rather
>>>   huge Shenandoah backport patch is an absolute no-go.
>>> 
>>>   This is an updated patch that includes numerous bugfixes and
>>>   improvements in Shenandoah. It is what Red Hat is going to ship
>>> in
>>>   their 11.0.8 release.
>>> 
>>>   Full webrev:
>>> 
>>> https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.04-all%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967016879&sdata=G1HjEJ68qFJ2uwRyxfFKNE8gcbsFXEdYi%2B9RyrUeG60%3D&reserved=0
>>> 
>>>   Webrev only for shared-code changes:
>>> 
>>> https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.04-shared%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=RrKNOvxc3%2FheThyi2cMiazE%2BxC9UfARxfTbmuoc640Y%3D&reserved=0
>>> 
>>>   Please let me know what you think.
>>> 
>>>   Roman
>>> 
>>>> On Fri, 2020-01-17 at 18:05 +0100, Roman Kennke wrote:
>>>> Hello,
>>>> 
>>>> The Shenandoah GC has been integrated into jdk12 a little over a
>>>> year
>>>> ago:
>>>> 
>>>> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fjdk%2Fjdk%2Frev%2F9c18c9d839d3&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=QJo2xwdHiEuaqsz26P2j%2BZdE6j3AxUvpnHVTjD9XGXI%3D&reserved=0
>>>> 
>>>> The Shenandoah team has been maintaining the Shenandoah-enabled
>>>> jdk11u
>>>> downstream repository since the inception of jdk11 under:
>>>> 
>>>> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fshenandoah%2Fjdk11&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=P%2FLkJ5VCrq4oWzik3WZ3GTy1a3mA52mxkITasawIMk0%3D&reserved=0
>>>> 
>>>> In order to make it more useful and accessible for users, we
>>>> would
>>>> like
>>>> to make it available in upstream jdk11u. The Shenandoah team at
>>>> Red
>>>> Hat
>>>> intends to maintain it the same way it had been maintained in
>>>> shenandoah/jdk11, in the course of regular jdk11u maintenance Red
>>>> Hat
>>>> already does.
>>>> 
>>>> Thanks to recent changes in Shenandoah over the last year, we are
>>>> now
>>>> able to fit the GC interface in jdk11 almost exactly. There are a
>>>> few
>>>> exceptions though. We tried to follow the following pattern for
>>>> all
>>>> shared-code changes where it made sense:
>>>> 
>>>> #if INCLUDE_SHENANDOAH_GC
>>>> if (UseShenandoahGC) {
>>>>   ...
>>>> }
>>>> #endif
>>>> 
>>>> This ensures that the Shenandoah-specific changes are cut out of
>>>> the
>>>> build when building without Shenandoah, and avoid those code
>>>> paths at
>>>> runtime when running without Shenandoah.
>>>> 
>>>> Also, there are a few places where this was not possible or where
>>>> it
>>>> didn't seem feasible, but we tried to keep them few and obvious.
>>>> Whenever this is the case, we are mirroring as close as possible
>>>> what
>>>> we
>>>> have in jdk/jdk.
>>>> 
>>>> Architecture-wise, Shenandoah runs on x86_64, aarch64, x86_32.
>>>> Other
>>>> architectures still build fine, and Shenandoah gets automatically
>>>> disabled during configure if not supported. OS-wise, Shenandoah
>>>> runs
>>>> on
>>>> Linux, Windows, and is known to run on MacOS X and Solaris too.
>>>> 
>>>> I wasn't sure which form this should take. I decided to start
>>>> with
>>>> the
>>>> easiest possible form this could take: a simple patch/webrev and
>>>> an
>>>> RFR.
>>>> Let me know if you need a JIRA issue or any other formalities to
>>>> track that.
>>>> 
>>>> Full webrev:
>>>> https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.02-all%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=eHspbGt5Eox4EMv8hstTBwsqFYefGNO2acGymP7ZfN0%3D&reserved=0
>>>> 
>>>> Webrev only for shared-code changes:
>>>> https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.02-shared%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=LGghbNRyHPSJLK8l2Oyhxh1lhD46LmvZtN1G5htYBpk%3D&reserved=0
>>>> 
>>>> The webrevs are based on and depend on the arraycopy patch
>>>> proposed
>>>> for
>>>> review here:
>>>> 
>>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openjdk.java.net%2Fpipermail%2Fjdk-updates-dev%2F2020-January%2F002396.html&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=Kw0qkfvtVNVrCiMeypwr4BHX9e3bZafJnVCMVY%2BvOOw%3D&reserved=0
>>>> 
>>>> Testing:
>>>> The code has been tested many many times, continuously while
>>>> maintaining
>>>> it. It has also served as the basis for RPMs and shipped to
>>>> customers.
>>>> We are running all sorts of tests of the hotspot/jtreg testsuite,
>>>> in
>>>> particular hotspot_gc_shenandoah, CTW tests, the regular tier*
>>>> tests,
>>>> several popular benchmarks (specjvm, specjbb on a regular basis),
>>>> on
>>>> a
>>>> nightly basis.
>>>> 
>>>> Please let me know what you think.
>>>> 
>>>> Thanks,
>>>> Roman
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>>

More information about the jdk-updates-dev mailing list