RFR (11u, XXL): Upstream/backport Shenandoah to JDK11u
Mario Torre
neugens at redhat.com
Thu Jul 9 09:44:00 UTC 2020
Hi Roman,
Can you please start a new thread to allow for a proper review?
(Whether approved by maintainer or not I think the patch will want a
review, right?)
We can continue this thread dedicated to the philosophical questions
about life, universe and backports :)
Cheers,
Mario
On Wed, Jul 8, 2020 at 8:32 PM Roman Kennke <rkennke at redhat.com> wrote:
>
> Hi Gil & all,
>
> I put in some extra work and had a deep look at the remaining unguarded
> changes in shared-code, with an eye to either eliminating them
> altogether, or - where this is not possible - guard them with build-
> time #if INCLUDE_SHENANDOAHGC or similar. The guiding principle in this
> effort has been that building without Shenandoah (--with-jvm-features=-
> shenandoahgc) would compile *the exact same* code that it does now. I
> believe I achieved this with the following proposed changes:
>
> Full webrev (incl. Shenandoah-only files):
> https://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.06-all/
>
> Shared-code-only changes:
> https://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.06-shared/
>
> The latter webrev exludes all files that are only compiled with
> Shenandoah enabled.
>
> Also, compared to previous webrevs, this excludes Shenandoah by
> default.
>
> As far as I can tell, this literally compiles the same code into
> libjvm.so as it does now, when building without Shenandoah. The only
> exception is the inclusion of the global flag UseShenandoahGC, which is
> by-design. When this is enabled in a build without Shenandoah, it
> prints the appropriate error message and exist (exactly like with any
> other GC that is not built-in).
>
> I've been thinking about if we can come up with a mechanical proof of
> correctness. The first best thing would have been reproducible builds,
> but we don't have that (and the UseShenandoahGC flag would mess it up
> too). The other option would be to dump preprocessed sources and diff
> them, but that is disturbed by the necessary inclusion of
> "utilities/macros.hpp" which gives us the INCLUDE_SHENANDOAHGC macro.
>
> Which leaves us with carefully inspecting the webrev by multiple sets
> of eyes and brains.
>
> Gil, would the change like this ease your concerns?
>
> Thank you,
> Roman
>
>
> On Fri, 2020-06-26 at 23:39 +0000, Gil Tene wrote:
> > As I noted before, we have serious reservations about adding major
> > new features to a
> > stable LTS that is upstream of all of 11u. Based on daily
> > interactions with tons of OpenJDK
> > end users, I can say that the vast majority of end users that are
> > ramping up on 11u are
> > demanding stability and maturity above all else. The addition of such
> > major changes in
> > an 11u update will cause many of them to stall updates, and
> > potentially stall the uptake
> > of 11u altogether, waiting for 11u to actually stabilize first
> > (evidenced by no longer doing
> > such things), so they can keep up with fixes and security issues
> > without having to take on
> > potential regressions driven by late added or experimental features.
> >
> > If you want a modern GA'ed OpenJDK with all the latest features in
> > it, use one. Please
> > please please don't force people who choose to stick with a given
> > OpenJDK version for
> > stability, and knowingly did not move to a later OpenJDK version, to
> > deploy the newly
> > implemented features that you like by pushing for back-porting them
> > to the upstream
> > project that they get their security updates from...
> >
> > I can see a potentially good case for a downstream-from-11u project
> > that incorporates
> > ongoing development (as opposed to bug fixes) from later upstream
> > versions, and would
> > be happy to collaborate on one. But the 11u that is upstream of all
> > other 11u's should not
> > be seen as a developer's playground, or a place to add cool new
> > features to a stable
> > release. The upstream versions are there for that purpose. The valid
> > interests of
> > deep-bench, self-supporting groups making use of OpenJDK, who can
> > take on the stability
> > risks involved due to the depth of their technical bench, should not
> > overcome the interests
> > of the millions of consumers of the 11u builds that do not have that
> > luxury, and who
> > need and expect stable (in the changing only where necessary sense)
> > 11u releases.
> >
> > I'd like to point out that 13u already includes Shenandoah. 15u
> > (expected in 3 months)
> > already includes Shenandoah as a non-experimental feature. And we can
> > all collaborate
> > on back-porting bug-fixes to those to updates to keep a lively
> > version of OpenJDK that
> > includes Shenandoah up to date from a bug fix and security update
> > point of view. For
> > those insisting on waiting for an LTS (on the assumption that LTSs
> > are long term stable
> > and don't get those sort of messing-with-after-the-fact done to them,
> > mind you), 17u will
> > be out in 15 months.
> >
> > So we are not talking about some far-away future for end users that
> > want Shenandoah.
> > It is here, now, in multiple consumable forms, and available for
> > people not looking for
> > the stability of an LTS, and even for people who do (as they can use
> > the Red Hat 11u
> > builds). It it will be available in more forms very soon with no
> > effort, and we don't have
> > to destabilize 11u for others to make that happen.
> >
> > — Gil.
> >
> > > On Jun 26, 2020, at 1:51 PM, Mathiske, Bernd <mathiske at amazon.com>
> > > wrote:
> > >
> > > Hi Roman,
> > >
> > > We are also in favor of upstreaming Shenandoah to jdk11u, for the
> > > same reasons Aditya listed, and we agree with everything else he
> > > has written. Similar to Aditya’s message below, we believe that
> > > Shenandoah has strong demand and potential for those running
> > > jdk11u. Targeting jdk11u is the most efficient way to reach a
> > > large user base and to enable production use of Shenandoah.
> > >
> > > At Amazon, we plan to continue to contribute to Shenandoah, and we
> > > fully intend to help maintain Shenandoah in jdk11u.
> > >
> > > Best,
> > >
> > > Bernd, Kelvin, Azeem
> > >
> > > On 6/26/20, 10:36 AM, "jdk-updates-dev on behalf of Aditya
> > > Mandaleeka" <jdk-updates-dev-retn at openjdk.java.net on behalf of
> > > adityam at microsoft.com> wrote:
> > >
> > > Hi Roman,
> > >
> > > I am in favor of this proposed upstreaming of Shenandoah into
> > > jdk11u. Microsoft
> > > has some long-term commitments to Java 11 customers and having
> > > Shenandoah
> > > included in jdk11u upstream would definitely be well-received by
> > > my team and by
> > > our customers. There is demand for this kind of GC option on
> > > OpenJDK 11, and it
> > > would be beneficial to the community for it to be available on
> > > the upstream
> > > OpenJDK rather than only in specific downstream releases.
> > > Coupled with the fact
> > > that Shenandoah has been tested on JDK 11 for quite a while now,
> > > I think this
> > > upstreaming makes sense.
> > >
> > > You mentioned in your original mail that your team at RedHat
> > > intends to maintain
> > > this proposed jdk11u version the same way it has been maintained
> > > in
> > > shenandoah/jdk11. That is great--from what I’ve seen in these
> > > past few months
> > > that I’ve been involved, I’ve been impressed by the way
> > > backports to
> > > shenandoah/jdk11 are being handled, the effort that is made to
> > > keep the
> > > Shenandoah code closely aligned between sh/11 and jdk/jdk, and
> > > the care taken to
> > > ensure that the impact on shared code is kept to a minimum.
> > >
> > > At Microsoft, we’ve already been contributing patches and
> > > investigating and
> > > reporting issues based on our testing of Shenandoah, both on
> > > jdk/jdk and
> > > shenandoah/jdk11. We intend to continue this kind of
> > > involvement, so if this
> > > upstreaming to jdk11u goes through, your team will also have our
> > > support in
> > > maintaining Shenandoah there.
> > >
> > > Thanks,
> > > Aditya
> > >
> > > -----Original Message-----
> > > From: jdk-updates-dev <jdk-updates-dev-retn at openjdk.java.net> On
> > > Behalf Of Roman Kennke
> > > Sent: Thursday, June 25, 2020 4:16 AM
> > > To: jdk-updates-dev at openjdk.java.net
> > > Subject: Re: RFR (11u, XXL): Upstream/backport Shenandoah to
> > > JDK11u
> > >
> > > I would like to revive this RFR, I believe we haven't come to a
> > > conclusion yet. I still think it would be very useful to have
> > > the
> > > Shenandoah GC included in jdk11u upstream. I have heard from
> > > several
> > > vendors who would like to ship Shenandoah, but maintaining the
> > > rather
> > > huge Shenandoah backport patch is an absolute no-go.
> > >
> > > This is an updated patch that includes numerous bugfixes and
> > > improvements in Shenandoah. It is what Red Hat is going to ship
> > > in
> > > their 11.0.8 release.
> > >
> > > Full webrev:
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.04-all%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967016879&sdata=G1HjEJ68qFJ2uwRyxfFKNE8gcbsFXEdYi%2B9RyrUeG60%3D&reserved=0
> > >
> > > Webrev only for shared-code changes:
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.04-shared%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=RrKNOvxc3%2FheThyi2cMiazE%2BxC9UfARxfTbmuoc640Y%3D&reserved=0
> > >
> > > Please let me know what you think.
> > >
> > > Roman
> > >
> > > > On Fri, 2020-01-17 at 18:05 +0100, Roman Kennke wrote:
> > > > Hello,
> > > >
> > > > The Shenandoah GC has been integrated into jdk12 a little over a
> > > > year
> > > > ago:
> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fjdk%2Fjdk%2Frev%2F9c18c9d839d3&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=QJo2xwdHiEuaqsz26P2j%2BZdE6j3AxUvpnHVTjD9XGXI%3D&reserved=0
> > > >
> > > > The Shenandoah team has been maintaining the Shenandoah-enabled
> > > > jdk11u
> > > > downstream repository since the inception of jdk11 under:
> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fshenandoah%2Fjdk11&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=P%2FLkJ5VCrq4oWzik3WZ3GTy1a3mA52mxkITasawIMk0%3D&reserved=0
> > > >
> > > > In order to make it more useful and accessible for users, we
> > > > would
> > > > like
> > > > to make it available in upstream jdk11u. The Shenandoah team at
> > > > Red
> > > > Hat
> > > > intends to maintain it the same way it had been maintained in
> > > > shenandoah/jdk11, in the course of regular jdk11u maintenance Red
> > > > Hat
> > > > already does.
> > > >
> > > > Thanks to recent changes in Shenandoah over the last year, we are
> > > > now
> > > > able to fit the GC interface in jdk11 almost exactly. There are a
> > > > few
> > > > exceptions though. We tried to follow the following pattern for
> > > > all
> > > > shared-code changes where it made sense:
> > > >
> > > > #if INCLUDE_SHENANDOAH_GC
> > > > if (UseShenandoahGC) {
> > > > ...
> > > > }
> > > > #endif
> > > >
> > > > This ensures that the Shenandoah-specific changes are cut out of
> > > > the
> > > > build when building without Shenandoah, and avoid those code
> > > > paths at
> > > > runtime when running without Shenandoah.
> > > >
> > > > Also, there are a few places where this was not possible or where
> > > > it
> > > > didn't seem feasible, but we tried to keep them few and obvious.
> > > > Whenever this is the case, we are mirroring as close as possible
> > > > what
> > > > we
> > > > have in jdk/jdk.
> > > >
> > > > Architecture-wise, Shenandoah runs on x86_64, aarch64, x86_32.
> > > > Other
> > > > architectures still build fine, and Shenandoah gets automatically
> > > > disabled during configure if not supported. OS-wise, Shenandoah
> > > > runs
> > > > on
> > > > Linux, Windows, and is known to run on MacOS X and Solaris too.
> > > >
> > > > I wasn't sure which form this should take. I decided to start
> > > > with
> > > > the
> > > > easiest possible form this could take: a simple patch/webrev and
> > > > an
> > > > RFR.
> > > > Let me know if you need a JIRA issue or any other formalities to
> > > > track that.
> > > >
> > > > Full webrev:
> > > > https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.02-all%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=eHspbGt5Eox4EMv8hstTBwsqFYefGNO2acGymP7ZfN0%3D&reserved=0
> > > >
> > > > Webrev only for shared-code changes:
> > > > https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.02-shared%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=LGghbNRyHPSJLK8l2Oyhxh1lhD46LmvZtN1G5htYBpk%3D&reserved=0
> > > >
> > > > The webrevs are based on and depend on the arraycopy patch
> > > > proposed
> > > > for
> > > > review here:
> > > >
> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openjdk.java.net%2Fpipermail%2Fjdk-updates-dev%2F2020-January%2F002396.html&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=Kw0qkfvtVNVrCiMeypwr4BHX9e3bZafJnVCMVY%2BvOOw%3D&reserved=0
> > > >
> > > > Testing:
> > > > The code has been tested many many times, continuously while
> > > > maintaining
> > > > it. It has also served as the basis for RPMs and shipped to
> > > > customers.
> > > > We are running all sorts of tests of the hotspot/jtreg testsuite,
> > > > in
> > > > particular hotspot_gc_shenandoah, CTW tests, the regular tier*
> > > > tests,
> > > > several popular benchmarks (specjvm, specjbb on a regular basis),
> > > > on
> > > > a
> > > > nightly basis.
> > > >
> > > > Please let me know what you think.
> > > >
> > > > Thanks,
> > > > Roman
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
>
--
Mario Torre
Associate Manager, Software Engineering
Red Hat GmbH <https://www.redhat.com>
9704 A60C B4BE A8B8 0F30 9205 5D7E 4952 3F65 7898
More information about the jdk-updates-dev
mailing list