RFR (11u, XXL): Upstream/backport Shenandoah to JDK11u
Roman Kennke
rkennke at redhat.com
Wed Jul 8 18:31:17 UTC 2020
Hi Gil & all,
I put in some extra work and had a deep look at the remaining unguarded
changes in shared-code, with an eye to either eliminating them
altogether, or - where this is not possible - guard them with build-
time #if INCLUDE_SHENANDOAHGC or similar. The guiding principle in this
effort has been that building without Shenandoah (--with-jvm-features=-
shenandoahgc) would compile *the exact same* code that it does now. I
believe I achieved this with the following proposed changes:
Full webrev (incl. Shenandoah-only files):
https://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.06-all/
Shared-code-only changes:
https://cr.openjdk.java.net/~rkennke/shenandoah-jdk11u-upstream/webrev.06-shared/
The latter webrev exludes all files that are only compiled with
Shenandoah enabled.
Also, compared to previous webrevs, this excludes Shenandoah by
default.
As far as I can tell, this literally compiles the same code into
libjvm.so as it does now, when building without Shenandoah. The only
exception is the inclusion of the global flag UseShenandoahGC, which is
by-design. When this is enabled in a build without Shenandoah, it
prints the appropriate error message and exist (exactly like with any
other GC that is not built-in).
I've been thinking about if we can come up with a mechanical proof of
correctness. The first best thing would have been reproducible builds,
but we don't have that (and the UseShenandoahGC flag would mess it up
too). The other option would be to dump preprocessed sources and diff
them, but that is disturbed by the necessary inclusion of
"utilities/macros.hpp" which gives us the INCLUDE_SHENANDOAHGC macro.
Which leaves us with carefully inspecting the webrev by multiple sets
of eyes and brains.
Gil, would the change like this ease your concerns?
Thank you,
Roman
On Fri, 2020-06-26 at 23:39 +0000, Gil Tene wrote:
> As I noted before, we have serious reservations about adding major
> new features to a
> stable LTS that is upstream of all of 11u. Based on daily
> interactions with tons of OpenJDK
> end users, I can say that the vast majority of end users that are
> ramping up on 11u are
> demanding stability and maturity above all else. The addition of such
> major changes in
> an 11u update will cause many of them to stall updates, and
> potentially stall the uptake
> of 11u altogether, waiting for 11u to actually stabilize first
> (evidenced by no longer doing
> such things), so they can keep up with fixes and security issues
> without having to take on
> potential regressions driven by late added or experimental features.
>
> If you want a modern GA'ed OpenJDK with all the latest features in
> it, use one. Please
> please please don't force people who choose to stick with a given
> OpenJDK version for
> stability, and knowingly did not move to a later OpenJDK version, to
> deploy the newly
> implemented features that you like by pushing for back-porting them
> to the upstream
> project that they get their security updates from...
>
> I can see a potentially good case for a downstream-from-11u project
> that incorporates
> ongoing development (as opposed to bug fixes) from later upstream
> versions, and would
> be happy to collaborate on one. But the 11u that is upstream of all
> other 11u's should not
> be seen as a developer's playground, or a place to add cool new
> features to a stable
> release. The upstream versions are there for that purpose. The valid
> interests of
> deep-bench, self-supporting groups making use of OpenJDK, who can
> take on the stability
> risks involved due to the depth of their technical bench, should not
> overcome the interests
> of the millions of consumers of the 11u builds that do not have that
> luxury, and who
> need and expect stable (in the changing only where necessary sense)
> 11u releases.
>
> I'd like to point out that 13u already includes Shenandoah. 15u
> (expected in 3 months)
> already includes Shenandoah as a non-experimental feature. And we can
> all collaborate
> on back-porting bug-fixes to those to updates to keep a lively
> version of OpenJDK that
> includes Shenandoah up to date from a bug fix and security update
> point of view. For
> those insisting on waiting for an LTS (on the assumption that LTSs
> are long term stable
> and don't get those sort of messing-with-after-the-fact done to them,
> mind you), 17u will
> be out in 15 months.
>
> So we are not talking about some far-away future for end users that
> want Shenandoah.
> It is here, now, in multiple consumable forms, and available for
> people not looking for
> the stability of an LTS, and even for people who do (as they can use
> the Red Hat 11u
> builds). It it will be available in more forms very soon with no
> effort, and we don't have
> to destabilize 11u for others to make that happen.
>
> — Gil.
>
> > On Jun 26, 2020, at 1:51 PM, Mathiske, Bernd <mathiske at amazon.com>
> > wrote:
> >
> > Hi Roman,
> >
> > We are also in favor of upstreaming Shenandoah to jdk11u, for the
> > same reasons Aditya listed, and we agree with everything else he
> > has written. Similar to Aditya’s message below, we believe that
> > Shenandoah has strong demand and potential for those running
> > jdk11u. Targeting jdk11u is the most efficient way to reach a
> > large user base and to enable production use of Shenandoah.
> >
> > At Amazon, we plan to continue to contribute to Shenandoah, and we
> > fully intend to help maintain Shenandoah in jdk11u.
> >
> > Best,
> >
> > Bernd, Kelvin, Azeem
> >
> > On 6/26/20, 10:36 AM, "jdk-updates-dev on behalf of Aditya
> > Mandaleeka" <jdk-updates-dev-retn at openjdk.java.net on behalf of
> > adityam at microsoft.com> wrote:
> >
> > Hi Roman,
> >
> > I am in favor of this proposed upstreaming of Shenandoah into
> > jdk11u. Microsoft
> > has some long-term commitments to Java 11 customers and having
> > Shenandoah
> > included in jdk11u upstream would definitely be well-received by
> > my team and by
> > our customers. There is demand for this kind of GC option on
> > OpenJDK 11, and it
> > would be beneficial to the community for it to be available on
> > the upstream
> > OpenJDK rather than only in specific downstream releases.
> > Coupled with the fact
> > that Shenandoah has been tested on JDK 11 for quite a while now,
> > I think this
> > upstreaming makes sense.
> >
> > You mentioned in your original mail that your team at RedHat
> > intends to maintain
> > this proposed jdk11u version the same way it has been maintained
> > in
> > shenandoah/jdk11. That is great--from what I’ve seen in these
> > past few months
> > that I’ve been involved, I’ve been impressed by the way
> > backports to
> > shenandoah/jdk11 are being handled, the effort that is made to
> > keep the
> > Shenandoah code closely aligned between sh/11 and jdk/jdk, and
> > the care taken to
> > ensure that the impact on shared code is kept to a minimum.
> >
> > At Microsoft, we’ve already been contributing patches and
> > investigating and
> > reporting issues based on our testing of Shenandoah, both on
> > jdk/jdk and
> > shenandoah/jdk11. We intend to continue this kind of
> > involvement, so if this
> > upstreaming to jdk11u goes through, your team will also have our
> > support in
> > maintaining Shenandoah there.
> >
> > Thanks,
> > Aditya
> >
> > -----Original Message-----
> > From: jdk-updates-dev <jdk-updates-dev-retn at openjdk.java.net> On
> > Behalf Of Roman Kennke
> > Sent: Thursday, June 25, 2020 4:16 AM
> > To: jdk-updates-dev at openjdk.java.net
> > Subject: Re: RFR (11u, XXL): Upstream/backport Shenandoah to
> > JDK11u
> >
> > I would like to revive this RFR, I believe we haven't come to a
> > conclusion yet. I still think it would be very useful to have
> > the
> > Shenandoah GC included in jdk11u upstream. I have heard from
> > several
> > vendors who would like to ship Shenandoah, but maintaining the
> > rather
> > huge Shenandoah backport patch is an absolute no-go.
> >
> > This is an updated patch that includes numerous bugfixes and
> > improvements in Shenandoah. It is what Red Hat is going to ship
> > in
> > their 11.0.8 release.
> >
> > Full webrev:
> >
> > https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.04-all%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967016879&sdata=G1HjEJ68qFJ2uwRyxfFKNE8gcbsFXEdYi%2B9RyrUeG60%3D&reserved=0
> >
> > Webrev only for shared-code changes:
> >
> > https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.04-shared%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=RrKNOvxc3%2FheThyi2cMiazE%2BxC9UfARxfTbmuoc640Y%3D&reserved=0
> >
> > Please let me know what you think.
> >
> > Roman
> >
> > > On Fri, 2020-01-17 at 18:05 +0100, Roman Kennke wrote:
> > > Hello,
> > >
> > > The Shenandoah GC has been integrated into jdk12 a little over a
> > > year
> > > ago:
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fjdk%2Fjdk%2Frev%2F9c18c9d839d3&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=QJo2xwdHiEuaqsz26P2j%2BZdE6j3AxUvpnHVTjD9XGXI%3D&reserved=0
> > >
> > > The Shenandoah team has been maintaining the Shenandoah-enabled
> > > jdk11u
> > > downstream repository since the inception of jdk11 under:
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fshenandoah%2Fjdk11&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=P%2FLkJ5VCrq4oWzik3WZ3GTy1a3mA52mxkITasawIMk0%3D&reserved=0
> > >
> > > In order to make it more useful and accessible for users, we
> > > would
> > > like
> > > to make it available in upstream jdk11u. The Shenandoah team at
> > > Red
> > > Hat
> > > intends to maintain it the same way it had been maintained in
> > > shenandoah/jdk11, in the course of regular jdk11u maintenance Red
> > > Hat
> > > already does.
> > >
> > > Thanks to recent changes in Shenandoah over the last year, we are
> > > now
> > > able to fit the GC interface in jdk11 almost exactly. There are a
> > > few
> > > exceptions though. We tried to follow the following pattern for
> > > all
> > > shared-code changes where it made sense:
> > >
> > > #if INCLUDE_SHENANDOAH_GC
> > > if (UseShenandoahGC) {
> > > ...
> > > }
> > > #endif
> > >
> > > This ensures that the Shenandoah-specific changes are cut out of
> > > the
> > > build when building without Shenandoah, and avoid those code
> > > paths at
> > > runtime when running without Shenandoah.
> > >
> > > Also, there are a few places where this was not possible or where
> > > it
> > > didn't seem feasible, but we tried to keep them few and obvious.
> > > Whenever this is the case, we are mirroring as close as possible
> > > what
> > > we
> > > have in jdk/jdk.
> > >
> > > Architecture-wise, Shenandoah runs on x86_64, aarch64, x86_32.
> > > Other
> > > architectures still build fine, and Shenandoah gets automatically
> > > disabled during configure if not supported. OS-wise, Shenandoah
> > > runs
> > > on
> > > Linux, Windows, and is known to run on MacOS X and Solaris too.
> > >
> > > I wasn't sure which form this should take. I decided to start
> > > with
> > > the
> > > easiest possible form this could take: a simple patch/webrev and
> > > an
> > > RFR.
> > > Let me know if you need a JIRA issue or any other formalities to
> > > track that.
> > >
> > > Full webrev:
> > > https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.02-all%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=eHspbGt5Eox4EMv8hstTBwsqFYefGNO2acGymP7ZfN0%3D&reserved=0
> > >
> > > Webrev only for shared-code changes:
> > > https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcr.openjdk.java.net%2F~rkennke%2Fshenandoah-jdk11u-upstream%2Fwebrev.02-shared%2F&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=LGghbNRyHPSJLK8l2Oyhxh1lhD46LmvZtN1G5htYBpk%3D&reserved=0
> > >
> > > The webrevs are based on and depend on the arraycopy patch
> > > proposed
> > > for
> > > review here:
> > >
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openjdk.java.net%2Fpipermail%2Fjdk-updates-dev%2F2020-January%2F002396.html&data=02%7C01%7Cadityam%40microsoft.com%7C7aec62734d22458243d908d818f974c5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637286806967026872&sdata=Kw0qkfvtVNVrCiMeypwr4BHX9e3bZafJnVCMVY%2BvOOw%3D&reserved=0
> > >
> > > Testing:
> > > The code has been tested many many times, continuously while
> > > maintaining
> > > it. It has also served as the basis for RPMs and shipped to
> > > customers.
> > > We are running all sorts of tests of the hotspot/jtreg testsuite,
> > > in
> > > particular hotspot_gc_shenandoah, CTW tests, the regular tier*
> > > tests,
> > > several popular benchmarks (specjvm, specjbb on a regular basis),
> > > on
> > > a
> > > nightly basis.
> > >
> > > Please let me know what you think.
> > >
> > > Thanks,
> > > Roman
> > >
> > >
> > >
> > >
> > >
> > >
> > >
More information about the jdk-updates-dev
mailing list