...

[Gil Tene]

> On Jun 29, 2020, at 2:56 PM, Mathiske, Bernd <mathiske at amazon.com <mailto:mathiske at amazon.com>> wrote:
> 
> From what I read, there seems to be consensus that the risk of the proposed RFR is in fact minimal,

No. Unless you consider "consensus" to not include my voice. Or e.g. Attila's.

I consider the risk large. I concede that a lot of admirable work has been
done to make it as small as possible given the magnitude of the change.
But this is a huge change no matter how you look at it. With significant risk.

> assuming deep review and other careful measures, as for the JFR 8u backport. So we are focusing on the reward side.

Sure. Some are focusing on the reward. I argue the rewards doesn't matter
in an already-out LTS.

> 
>>> Necessity is a binary bar.
> Don’t we all always ask “how necessary” or “necessary for what”? Isn’t necessity simply a strong reading on the reward meter? And, indeed, we ask “necessary for whom”.

True. The necessity and level of detail we are willing to go into when
fixing crashing bugs can be argued too. But at least there we are fixing
something that is actually broken.

> 
>>> This list tends to be dominated by developers of OpenJDK.
>>> I claim that our community (for the updates projects) is the *users* of OpenJDK.
> 
> The developer minority that is writing to each other here has significant exposure to user opinions and issues.
> 
> Aren’t average and tail latencies wide-spread user pain points?

Not for stable production systems that are not looking for change, and
just want to keep up with security issues and stability bugs. And that
describes the vast majority of deployments of a stable LTS.

Suffering existing unhappy metric behavior that has been there
all along cannot be compared taking on regression in a stable
production system. they don't live in the same operational realm.

If the proposed features help with such unhappy pain points, users
that look to address them and become happier can already do so by
changing to new versions or switching to other distros within the
same version that include the desired feature..


> Shouldn’t we proliferate concurrent GC to address that? And isn’t upgrading to another JDK release also a huge user pain point, at least for some users?

OpenJDK users that expect stability and security updates have
no other place to get them. Forcing all of them to accept the risk of
changes to serve the wishes of people who do have plenty of other
options (move to newer versions, move to other downstream distros
that choose to add features) seems like an obvious wrong to me.

There are downstream distros that are comfortable with feature
differentiation and during-LTS changes that they can support and
deal with. As you know, we (at Azul) are VERY comfortable running
downstream distros with significant feature differences (e.g. Zing, where
we use a hotspot-express-like model), and we are not alone in that:
The Red Hat team is clearly comfortable with the same (to them and to
their downstream users, this specific item is not a feature
change, as they already ship with it). The Twitter team is clearly very
comfortable doing that, and so is the DragonWell team. And the SAP
team. And (I think) the Google team too. Each of these teams (and their
downstream users) have chosen to take on the burden of feature
improvements and changes and the risk involved,. It's their choice.

But there are distros that do not aim for feature differentiation or rapid
feature development, and simply focus of stability and keeping up with  security
updates, leaving new features to new versions of OpenJDK, and incorporating
only things that are know to be either in the [same-version] upstream or
expected to get there quite soon (and with high confidence).
Our Zulu distro is one of those, and is massively used for that reason
and purpose. AOJ seems to be another. And I can't speak for Liberica, but
I suspect they are in this group as well.

I'm not quite sure where to place Corretto, as I had expected it to fit in the latter
group due to the public (non-Amazon-internal) consumption of the distro,
But at least some of the Corretto team is obviously on the side of
lets-add-features-to-the-LTS-if-they-are-interesting-enough, and perhaps the
huge Amazon-internal use and exposure is what makes the team comfortable
with that.

For the downstream consumers of the "stay close to vanilla" distros,
forcing a feature change in the middle of an LTS's life is counter to what
the release (and the distro) they chose to use and stay with is all about.

This upstream 11u project is where all these distros coordinate. It is the
common denominator. It is where 11u gets stabilized and maintained.
Not developed. And the needs of some of the distros require the common
denominator of the LTS to not introduce new features without a driving
necessity, not matter how cool. The users of those distros, IMO, represent
the vast majority of actual production use of OpenJDK in the field.

Cool new features simply don't belong in the stable, maintenance-mode
upstream LTS updates.  They belong in new OpoenJDK versions and in
downstream distros willing (and who's users are willing) to accept the
implications of during-LTS feature additions. Let's keep the common
denominator common, simple, and boring.

Arguing for "middle ground" between stability and innovation in an upstream
place that is expected (by many of its downstream users) to be in stabilization
and maintenance mode will create a a need for a different common place for
stable distros who don't seek change. I truly believe that non of us want that.
Some of us just seem to think that we can convince all other OpenJDK users
to give up on the "stay boring" approach to stability and maintenance. And I'm
just a stubborn voice reminding you that it's probably not our choice to make.

> See Goetz’s email for an existence proof, but there are many other cases in other companies. Isn’t sticking to LTS versions what users normally do? My experience is that, whether justified or not, most do just that. If you put these aspects together, the necessity to do something about Shenandoah sooner rather than later reveals itself. Individual examples of users to whom this does not apply do not change this. In fact, there are often enough good reasons to not use concurrent GC, depending on the use case. But without concurrent GC we have a serious gap in our ability to run latency-sensitive workloads right now, and we need to fix that.
> 
> BTW, we were under the impression that Azul fully supports the idea of backporting Shenandoah.
> https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-July/009808.html <https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-July/009808.html>

As I noted elsewhere, to me this (including Aarch64 in 8u) fits as an example of
the “gut wrenching” necessary category, and IMO meets and passes the bar of
"if we don't do this, the LTS will break and millions will be affected".

As with my prior containers example in 8u192, the already-here approachability
and affordability of Aarch64 cloud instances makes and already-here thing. And
the clear and already-here-or-near-term-coming desktop Aarch64 wave is also a
sign here. Backpoting the upstream MSFT contribution for Windows 10 Aarch64
targets to 8u and 11u will likely make sense for this reason. And Macs...

> 
> Looking for preceding guidance on the matter, there is this email thread (spread over two months).
> https://mail.openjdk.java.net/pipermail/jdk-updates-dev/2019-October/002053.html <https://mail.openjdk.java.net/pipermail/jdk-updates-dev/2019-October/002053.html>
> https://mail.openjdk.java.net/pipermail/jdk-updates-dev/2019-November/002075.html <https://mail.openjdk.java.net/pipermail/jdk-updates-dev/2019-November/002075.html>

This is not a case of a feature addition. JFR was part of Java 8 from the start, as
an integral part of the most common free Java 8 distro until sometime last year.
It then was temporarily (and to many quite annoyingly) removed because it was
just plain missing in OpenJDK 8u, making it harder to adopt for Oracle shops.
Downstream 8u distros (like Zulu and Dragonwell) have closed that gap a long
time ago in anticipation of eventual upstream merges, and the gap will finally be
closed in 8u itself soon.

> 
> I heard that at the Committer’s Workshop session on update releases in February, there was consensus that “features” should only be backported to an update release if they are already in use and maintained by at least two other major contributors to the updates project.

Again, that depends on what you call "consensus". The interests of the
massive numbers of users of stable LTSs is not addressed by that sort of
consensus. Having committed people able to readily and heroically fix
problems that arise after the fact does nothing for the stability concerns of
people who have to apply security updates but cannot afford the problems
that invariably arise when innovation is allowed to be bundled with them.

We had an unfortunate time overlap between two session at the Committer’s
Workshop that dealt with the same high level problem in different ways. At the
time this seaming consensus was being arrived at, I was in another room,
presenting on the value of OpenJDK MTS update releases (13u, 15u, with
continued bug fix and security updates until 18 months after the next LTS
comes out) as a means for dealing with the very same need; enabling practical
production adoption of new features; but without the need to accept feature
backports into LTSs.

— Gil.

> Furthermore, at that point, it becomes a big waste of resources to stay in sync, and by definition there already is a support commitment. Regarding the RFR at hand, there is more than the necessary commitment in this very email thread.
> 
> Bernd