RFR (11u, XXL): Upstream/backport Shenandoah to JDK11u

[Attila Szegedi]

I agree with everything Gil said and want to provide some additional perspective as a current operator of business-critical infrastructure relying on Java.

TL;DR: 

I’m currently operating a largeish cluster of machines running Java 11. I definitely wouldn't appreciate 11 acquiring new major features, flag-activated or not. I find it natural that if we want new features, we need to migrate to a new feature release of Java.

Longer version:

At my current job we transitioned from using Java 8 to Java 11 as our runtime maybe about a year ago. As expected, there were some minor adjustments to make, mostly due to some 3rd party libraries’ use of unsafe features or unwanted interference with the module system.

So, we’re currently _mostly_ happy campers on Java 11. We’d like to try the new low-latency GCs though, both ZGC and Shenandoah (enable both of them in our cluster of many identical nodes all running the same code) and see which one does better for our workloads. The logical way to do this for us in our mind was always to switch to Java 14. I never would consider having a “Java 11, but with Shenandoah” for this purpose. We understand that LTS is there for stability, and if we want new features, well… that’s what feature releases are for. 

“Hiding behind a flag” is also an irrelevant argument. Integration of a new GC algorithm is almost always a change that requires all kinds of plumbing changes to other parts of the runtime and those will be in active code paths, flag or not. 

As a closing opinionated thought, I think the OpenJDK developer community developed (hah) a quite severe feature-backporting habit mostly out of difficulties of making the Java 8-to-9 barrier with its introduction of modules, so Java 8 became a prime backporting target. It seems that habit, once established, now extends to 11 as well. I also believe this habit is misguided; there’s simply no similar technical hurdle upgrading from 11 to, say, 14 as there was from 8 to 9. (Well, I will allow that it is possible I am not aware of some issue that makes it hard.) I know some people get cold feet thinking of Oracle’s licensing terms for non-LTS versions feeling that forces them into lockstep upgrades if they adopt a non-LTS version at any given time, but that’s not much of an issue either as nowadays you don’t have to use Oracle distributions of Java and there’s plenty of quality alternative OpenJDK-based distributions that don’t come with these licensing terms attached.

Attila.

> On 2020. Jun 28., at 16:22, Gil Tene <gil at azul.com> wrote:
> 
> 
> 
>> On Jun 27, 2020, at 12:50 AM, Roman Kennke <rkennke at redhat.com> wrote:
>> 
>> Hi Gil,
>> 
>> I understand your concerns.
>> 
>> I wouldn't propose this if Shenandoah was actually an
>> experimental/playground feature. The contrary is the case - it is very
>> solid and ready for production use (and infact *in* production use by
>> Red Hat customers and others). It has received tons of testing by us
>> and others.
> 
> I am in no way challenging or debating the effort and testing put into it,
> or the good practices and experience built up. I'm pointing out that
> such major features should simply not be added to a stable LTS release.
> 
> The "playground" comment is not about the maturity of the feature being
> proposed. It is about the maturity of the release it is being proposed for.
> Adding features to a stable LTS train without a forcing necessity is
> the "treating it as a playground" I was talking about.
> 
> I know a whole bunch of OpenJDK developers here want this. And a lot
> of external people that like to work and play with leading edge stuff want
> it (on 11u) too. And I don't doubt that there are tens of thousands of people
> that would be happy to see it added.
> 
> I'm pointing out that there will be (literally) Millions of people affected
> who do not share this view, are not here on these lists speaking out
> about it, and for whom 11u LTS builds are a critical infrastructure
> competent where stability matters most. Those are the people who
> expect LTS releases to only have bug fixes and security updates applied.
> They represent the vast majority of production consumption of LTS builds.
> 
> And yes, we occasionally have to go beyond bug fixes and security
> updates, and add significant features that did not previously exist in
> that LTS version of Java or OpenJDK in any previous form. When
> we (very reluctantly) do that, it needs to be as a result of necessity.
> 
> IMO the trigger we should be looking at as we gauge necessity is
> "If we don't do this, the LTS will likely break in some way, and many
> (Millions?) will be affected"
> 
> A current gut-wrenching example of necessity is TLS 1.3 support in
> OpenJDK 8u: Java 8 will need to live long past the point where
> TLS 1.2 may be disallowed by some organizations, and the pressure
> is already beginning to show. Adding TLS 1.3 to Java 8 is an unavoidable
> necessity. The Java SE 8 spec was modified to enable TLS 1.3 support,
> and the externally-driven need is pretty clear. OpenJDK 8u cannot stay
> behind on this. So this qualifies. If we don't do this, OpenJDK 8u will
> break. Millions will be negatively affected.
> 
> A smaller example was the addition of container resource limit awareness
> in 8u192. The emergence of containers as common deployment
> environments and the friction between the JDK behavior and resource
> limit enforcement (throttling/thrashing, OOM killing) created painful enough
> situations that it was necessary to add a feature to the JDK implementation
> to allow it to live in acceptable ways in modern hardware deployments.
> So this one qualified too: If it were not done, 8u would have remained
> broken on container environments. Millions would have  been negatively
> affected.
> 
> We simply don't have necessity behind the suggestion for adding
> a brand new garbage collector to the upstream 11u distribution. We have
> people who'd like to see it happen, and people who will agree to deal with
> he issues that may arise and help fix them in future updates. And we have
> multiple groups that are willing to run it internally within their large
> organizations and self-support if needed. But we don't have an external
> forcing event that says that unless we do this we'll be causing tons of
> end-users to break or start malfunctioning.
> 
> [And yes, I'm sure some examples of unnecessary things added
> mid-life of a stable release in the past can be dug up. Nothing is pure
> in the past. But I doubt anything like "add a new collector to the mature
> upstream production release" can be found int the last 10 or so years]
> 
>> 
>> If a downstream vendor doesn't want to ship it, it is easily solved by
>> --with-jvm-features=-shenandoahgc. As I mentioned in my original email,
>> we make sure that we exclude Shenandoah-added code using #ifdef
>> INCLUDE_SHENANDOAHGC whereever reasonably possible. It doesn't zero the
>> risk, but it brings it pretty close.
> 
> This is not about how carefully this is applied, or how much "this shouldn't
> break anything, by design, really, and we tested it a lot" we can point to.
> 
> The best way to not break things is to not change them, Every experienced
> IT operator knows this instinctively, and a huge % of those choose to only use
> LTS versions (and wait 1+ years for them to stabilize before doing so)
> for that very reason. Those people represent a huge portion of the
> "customer" consuming the 11u updates this project builds.
> 
> Just imagine trying to explain this to a room full of SREs or sysadmins,
> trying to think of  a comeback to "Wait! So you are saying that I can't get
> security updates for 11u any more without accepting that they come with
> the code changes you made for adding in a brand new garbage collector?"
> 
> A "we worked hard to minimize actual code change wherever reasonably
> possible" answer will not fly in that room.
> 
>> 
>> I don't think a downstream jdk11+ or so project/repository would be all
>> that useful. I don't see many other such features to go into jdk11u, so
>> it might well be a Shenandoah-only thing - which we already have in the
>> form of shenandoah/jdk11 repository, which is very well maintained
>> downstream from jdk11u + Shenandoah.
> 
> I think this is one (big) example, and that we will likely face more of the same
> choices due to similar motivations in the years ahead, with useful features
> big and small looking to be back-ported, and with those wishes hitting
> against the primary purpose of an upstream LTS, and against it's main mission
> and useful role for the vast majority of downstream consumers.
> 
> I actually do think (and like) the idea of a downstream-from-11u thing that
> can take on new features can work (hotspot-expres-like or not). I would be
> interested in collaborating on such a thing, and think that it will have many
> interested consumers. And  there are several interesting improvements I would
> like to see added to an 11u-that-takes-on-new-feature thing if such a delivery
> vehicle was available. But I obviously don't think that thing and the upstream
> stable 11u that is all other 11u's update from should be conflated.
> 
>> 
>> If our current backporting practice is a concern I could offer to
>> tighten it up and only backport critical bugfixes, no feature
>> backports, no non-critical bugfixes, etc.
> 
> This is a feature backport. What happens after is not the issue. And
> it will be hard to argue for no additional increments after we add in an
> entire new collector that didn't previously exist in the release.
> 
>> 
>> That being said, I do understand your point. This is not about somehow
>> making my (or Red Hat's) life easier (it wouldn't - backporting to
>> jdk11u proper seems more bureaucratic than backporting to our own
>> downstream repo). My intention was to avoid fragmentation between
>> vendors. If we cannot get to an agreement here, I'd be fine to keep
>> maintaining shenandoah/jdk11 as we already do, and suggest that vendors
>> who want Shenandoah in their jdk11u builds to use that as a basis for
>> their builds.
> 
> That to me seems like the reasonable way to go, as long as no other
> downstream-from-11u thing that is better (as in allows multiple of these
> major feature backports to be consolidated togther) exists.
> 
> Downstream distros that want Shenandoah can be downstream from
> shenandoah/jdk11, as Red Hat [presumably] currently is. And those who
> want both can do both.
> 
>> *I* don't think that is the preferable solution.
> 
> And clearly respectfully disagree with each other on that.
> 
> I, for one, look forward to seeing the non-experimental Shenandoah
> in 15u come out, and will work to get people to use 15u in production
> as soon as possible (including making sure it keeps getting updated
> long enough to last for 18 months past the first 17u release). IMO that
> is the best path for driving new feature adoption. Unline in 11u, doing
> fixes to Shenandoah on 15u as we keep 15u up to date with bug fixes
> and security updates (including back-ports from later releases) will
> make perfect sense, as it will not be a late added feature.
>> 
>> Best regards,
>> Roman
>> 
>> 
>> 
>> On Fri, 2020-06-26 at 23:39 +0000, Gil Tene wrote:
>>> As I noted  before, we have serious reservations about adding major
>>> new features to a
>>> stable LTS that is upstream of all of 11u. Based on daily
>>> interactions with tons of OpenJDK
>>> end users, I can say that the vast majority of end users that are
>>> ramping up on 11u are
>>> demanding stability and maturity above all else. The addition of such
>>> major changes in
>>> an 11u update will cause many of them to stall updates, and
>>> potentially stall the uptake
>>> of 11u altogether, waiting for 11u to actually stabilize first
>>> (evidenced by no longer doing
>>> such things), so they can keep up with fixes and security issues
>>> without having to take on
>>> potential regressions driven by late added or experimental features.
>>> 
>>> If you want a modern GA'ed OpenJDK with all the latest features in
>>> it, use one. Please
>>> please please don't force people who choose to stick with a given
>>> OpenJDK version for
>>> stability, and knowingly did not move to a later OpenJDK version, to
>>> deploy the newly
>>> implemented features that you like by pushing for back-porting them
>>> to the upstream
>>> project that they get their security updates from...
>>> 
>>> I can see a potentially good case for a downstream-from-11u project
>>> that incorporates
>>> ongoing development (as opposed to bug fixes) from later upstream
>>> versions, and would
>>> be happy to collaborate on one. But the 11u that is upstream of all
>>> other 11u's should not
>>> be seen as a developer's playground, or a place to add cool new
>>> features to a stable
>>> release. The upstream versions are there for that purpose. The valid
>>> interests of
>>> deep-bench, self-supporting groups making use of OpenJDK, who can
>>> take on the stability
>>> risks involved due to the depth of their technical bench, should not
>>> overcome the interests
>>> of the millions of consumers of the 11u builds that do not have that
>>> luxury, and who
>>> need and expect stable (in the changing only where necessary sense)
>>> 11u releases.
>>> 
>>> I'd like to point out that 13u already includes Shenandoah. 15u
>>> (expected in 3 months)
>>> already includes Shenandoah as a non-experimental feature. And we can
>>> all collaborate
>>> on back-porting bug-fixes to those to updates to keep a lively
>>> version of OpenJDK that
>>> includes Shenandoah up to date from a bug fix and security update
>>> point of view. For
>>> those insisting on waiting for an LTS (on the assumption that LTSs
>>> are long term stable
>>> and don't get those sort of messing-with-after-the-fact done to them,
>>> mind you), 17u will
>>> be out in 15 months.
>>> 
>>> So we are not talking about some far-away future for end users that
>>> want Shenandoah.
>>> It is here, now, in multiple consumable forms, and available for
>>> people not looking for
>>> the stability of an LTS, and even for people who do (as they can use
>>> the Red Hat 11u
>>> builds). It it will be available in more forms very soon with no
>>> effort, and we don't have
>>> to destabilize 11u for others to make that happen.
>>> 
>>> — Gil.
>