RFR (11u, XXL): Upstream/backport Shenandoah to JDK11u

[Gil Tene]

> On Jun 27, 2020, at 12:50 AM, Roman Kennke <rkennke at redhat.com> wrote:
> 
> Hi Gil,
> 
> I understand your concerns.
> 
> I wouldn't propose this if Shenandoah was actually an
> experimental/playground feature. The contrary is the case - it is very
> solid and ready for production use (and infact *in* production use by
> Red Hat customers and others). It has received tons of testing by us
> and others.

I am in no way challenging or debating the effort and testing put into it,
or the good practices and experience built up. I'm pointing out that
such major features should simply not be added to a stable LTS release.

The "playground" comment is not about the maturity of the feature being
proposed. It is about the maturity of the release it is being proposed for.
Adding features to a stable LTS train without a forcing necessity is
the "treating it as a playground" I was talking about.

I know a whole bunch of OpenJDK developers here want this. And a lot
of external people that like to work and play with leading edge stuff want
it (on 11u) too. And I don't doubt that there are tens of thousands of people
that would be happy to see it added.

I'm pointing out that there will be (literally) Millions of people affected
who do not share this view, are not here on these lists speaking out
about it, and for whom 11u LTS builds are a critical infrastructure
competent where stability matters most. Those are the people who
expect LTS releases to only have bug fixes and security updates applied.
They represent the vast majority of production consumption of LTS builds.

And yes, we occasionally have to go beyond bug fixes and security
updates, and add significant features that did not previously exist in
that LTS version of Java or OpenJDK in any previous form. When
we (very reluctantly) do that, it needs to be as a result of necessity.

IMO the trigger we should be looking at as we gauge necessity is
"If we don't do this, the LTS will likely break in some way, and many
(Millions?) will be affected"

A current gut-wrenching example of necessity is TLS 1.3 support in
OpenJDK 8u: Java 8 will need to live long past the point where
TLS 1.2 may be disallowed by some organizations, and the pressure
is already beginning to show. Adding TLS 1.3 to Java 8 is an unavoidable
necessity. The Java SE 8 spec was modified to enable TLS 1.3 support,
and the externally-driven need is pretty clear. OpenJDK 8u cannot stay
behind on this. So this qualifies. If we don't do this, OpenJDK 8u will
break. Millions will be negatively affected.

A smaller example was the addition of container resource limit awareness
in 8u192. The emergence of containers as common deployment
environments and the friction between the JDK behavior and resource
limit enforcement (throttling/thrashing, OOM killing) created painful enough
situations that it was necessary to add a feature to the JDK implementation
to allow it to live in acceptable ways in modern hardware deployments.
So this one qualified too: If it were not done, 8u would have remained
broken on container environments. Millions would have  been negatively
affected.

We simply don't have necessity behind the suggestion for adding
a brand new garbage collector to the upstream 11u distribution. We have
people who'd like to see it happen, and people who will agree to deal with
he issues that may arise and help fix them in future updates. And we have
multiple groups that are willing to run it internally within their large
organizations and self-support if needed. But we don't have an external
forcing event that says that unless we do this we'll be causing tons of
end-users to break or start malfunctioning.

[And yes, I'm sure some examples of unnecessary things added
mid-life of a stable release in the past can be dug up. Nothing is pure
in the past. But I doubt anything like "add a new collector to the mature
upstream production release" can be found int the last 10 or so years]

> 
> If a downstream vendor doesn't want to ship it, it is easily solved by
> --with-jvm-features=-shenandoahgc. As I mentioned in my original email,
> we make sure that we exclude Shenandoah-added code using #ifdef
> INCLUDE_SHENANDOAHGC whereever reasonably possible. It doesn't zero the
> risk, but it brings it pretty close.

This is not about how carefully this is applied, or how much "this shouldn't
break anything, by design, really, and we tested it a lot" we can point to.

The best way to not break things is to not change them, Every experienced
IT operator knows this instinctively, and a huge % of those choose to only use
LTS versions (and wait 1+ years for them to stabilize before doing so)
for that very reason. Those people represent a huge portion of the
"customer" consuming the 11u updates this project builds.

Just imagine trying to explain this to a room full of SREs or sysadmins,
trying to think of  a comeback to "Wait! So you are saying that I can't get
security updates for 11u any more without accepting that they come with
the code changes you made for adding in a brand new garbage collector?"

A "we worked hard to minimize actual code change wherever reasonably
possible" answer will not fly in that room.

> 
> I don't think a downstream jdk11+ or so project/repository would be all
> that useful. I don't see many other such features to go into jdk11u, so
> it might well be a Shenandoah-only thing - which we already have in the
> form of shenandoah/jdk11 repository, which is very well maintained
> downstream from jdk11u + Shenandoah.

I think this is one (big) example, and that we will likely face more of the same
choices due to similar motivations in the years ahead, with useful features
big and small looking to be back-ported, and with those wishes hitting
against the primary purpose of an upstream LTS, and against it's main mission
and useful role for the vast majority of downstream consumers.

I actually do think (and like) the idea of a downstream-from-11u thing that
can take on new features can work (hotspot-expres-like or not). I would be
interested in collaborating on such a thing, and think that it will have many
interested consumers. And  there are several interesting improvements I would
like to see added to an 11u-that-takes-on-new-feature thing if such a delivery
vehicle was available. But I obviously don't think that thing and the upstream
stable 11u that is all other 11u's update from should be conflated.

> 
> If our current backporting practice is a concern I could offer to
> tighten it up and only backport critical bugfixes, no feature
> backports, no non-critical bugfixes, etc.

This is a feature backport. What happens after is not the issue. And
it will be hard to argue for no additional increments after we add in an
entire new collector that didn't previously exist in the release.

> 
> That being said, I do understand your point. This is not about somehow
> making my (or Red Hat's) life easier (it wouldn't - backporting to
> jdk11u proper seems more bureaucratic than backporting to our own
> downstream repo). My intention was to avoid fragmentation between
> vendors. If we cannot get to an agreement here, I'd be fine to keep
> maintaining shenandoah/jdk11 as we already do, and suggest that vendors
> who want Shenandoah in their jdk11u builds to use that as a basis for
> their builds.

That to me seems like the reasonable way to go, as long as no other
downstream-from-11u thing that is better (as in allows multiple of these
major feature backports to be consolidated togther) exists.

Downstream distros that want Shenandoah can be downstream from
shenandoah/jdk11, as Red Hat [presumably] currently is. And those who
want both can do both.

> *I* don't think that is the preferable solution.

And clearly respectfully disagree with each other on that.

I, for one, look forward to seeing the non-experimental Shenandoah
in 15u come out, and will work to get people to use 15u in production
as soon as possible (including making sure it keeps getting updated
long enough to last for 18 months past the first 17u release). IMO that
is the best path for driving new feature adoption. Unline in 11u, doing
fixes to Shenandoah on 15u as we keep 15u up to date with bug fixes
and security updates (including back-ports from later releases) will
make perfect sense, as it will not be a late added feature.
> 
> Best regards,
> Roman
> 
> 
> 
> On Fri, 2020-06-26 at 23:39 +0000, Gil Tene wrote:
>> As I noted  before, we have serious reservations about adding major
>> new features to a
>> stable LTS that is upstream of all of 11u. Based on daily
>> interactions with tons of OpenJDK
>> end users, I can say that the vast majority of end users that are
>> ramping up on 11u are
>> demanding stability and maturity above all else. The addition of such
>> major changes in
>> an 11u update will cause many of them to stall updates, and
>> potentially stall the uptake
>> of 11u altogether, waiting for 11u to actually stabilize first
>> (evidenced by no longer doing
>> such things), so they can keep up with fixes and security issues
>> without having to take on
>> potential regressions driven by late added or experimental features.
>> 
>> If you want a modern GA'ed OpenJDK with all the latest features in
>> it, use one. Please
>> please please don't force people who choose to stick with a given
>> OpenJDK version for
>> stability, and knowingly did not move to a later OpenJDK version, to
>> deploy the newly
>> implemented features that you like by pushing for back-porting them
>> to the upstream
>> project that they get their security updates from...
>> 
>> I can see a potentially good case for a downstream-from-11u project
>> that incorporates
>> ongoing development (as opposed to bug fixes) from later upstream
>> versions, and would
>> be happy to collaborate on one. But the 11u that is upstream of all
>> other 11u's should not
>> be seen as a developer's playground, or a place to add cool new
>> features to a stable
>> release. The upstream versions are there for that purpose. The valid
>> interests of
>> deep-bench, self-supporting groups making use of OpenJDK, who can
>> take on the stability
>> risks involved due to the depth of their technical bench, should not
>> overcome the interests
>> of the millions of consumers of the 11u builds that do not have that
>> luxury, and who
>> need and expect stable (in the changing only where necessary sense)
>> 11u releases.
>> 
>> I'd like to point out that 13u already includes Shenandoah. 15u
>> (expected in 3 months)
>> already includes Shenandoah as a non-experimental feature. And we can
>> all collaborate
>> on back-porting bug-fixes to those to updates to keep a lively
>> version of OpenJDK that
>> includes Shenandoah up to date from a bug fix and security update
>> point of view. For
>> those insisting on waiting for an LTS (on the assumption that LTSs
>> are long term stable
>> and don't get those sort of messing-with-after-the-fact done to them,
>> mind you), 17u will
>> be out in 15 months.
>> 
>> So we are not talking about some far-away future for end users that
>> want Shenandoah.
>> It is here, now, in multiple consumable forms, and available for
>> people not looking for
>> the stability of an LTS, and even for people who do (as they can use
>> the Red Hat 11u
>> builds). It it will be available in more forms very soon with no
>> effort, and we don't have
>> to destabilize 11u for others to make that happen.
>> 
>> — Gil.