11u testing (was: Re: [11u] RFR: 8258414: OldObjectSample events too expensive)

[Thomas Stüfe]

On Fri, Apr 23, 2021 at 4:32 PM Jaroslav Bachorík <
jaroslav.bachorik at datadoghq.com> wrote:

> Hi,
>
> On Thu, Apr 22, 2021 at 7:19 AM Thomas Stüfe <thomas.stuefe at gmail.com>
> wrote:
> >
> > Hi guys,
> >
> > Errors happen, that is normal. But I wonder if we need some clearer
> standards for testing downported patches? Currently the only requirement is
> to run tier1 tests, but nothing is mentioned about the configuration and
> the platforms to be tested. 11u is a stable maintenance release and in
> theory it should have a higher, not a lower bar for testing than the head
> release.
>
> I totally agree. This was my fault as I was lulled to a false sense of
> safety by the GH actions which I completely forgot were not happening
> for 11u.
>
> >
> > Surely testing just release is not enough, the debug builds need to be
> tested too. Then, arguably one should test on more than one platform. It's
> fine for individual contributors which don't have the resources, but larger
> companies should be able to test 11u patches on multiple platforms.
> Otherwise the burden of running the full gamut of tests and analysing
> accumulated errors lies overly by the 11u maintainers.
>
> IMO, a requirement to have a full spectrum of OS-HW combinations
> covered before pushing a backport will basically mean that the
> contributors without access to such resources will not be able/allowed
> to push backports at all.
> I am not saying it is wrong - just flagging the possible result.
>
>
Honestly, I would give preference to the need of the users regarding the
stability of the code base over the need of individual contributors to
backport a specific feature.

For the mainline JDK, we have the full power of the Oracle team, plus
Oracle's large testing infrastructure, plus a generous cooking time until a
release is rolled out, plus - lets be honest - a much longer time until
serious mass adoption.

For backport releases, we have a rather small team of maintainers and
reviewers, and testing mainly done by Red Hat and SAP? I may forget someone
but I am not aware of large scale testing done by others. But the way to
mass adoption on older releases is way shorter. A comparatively quick
baking time, then an update gets rolled out to millions of customers which
expect everything to continue smoothly. This is fine for a slow trickle of
important patches, but gets overwhelmed when facing many changes,
especially if they are of low quality. Therefore I think backport authors
should be aware of this and test their patches thoroughly before turning
them in. What constitutes "thorough" testing is up to the individual
author, and depends on the patch oc.

Cheers, Thomas