11u testing (was: Re: [11u] RFR: 8258414: OldObjectSample events too expensive)

[Mario Torre]

On Fri, Apr 23, 2021 at 4:33 PM Jaroslav Bachorík
<jaroslav.bachorik at datadoghq.com> wrote:
>
> Hi,
>
> On Thu, Apr 22, 2021 at 7:19 AM Thomas Stüfe <thomas.stuefe at gmail.com> wrote:
> >
> > Hi guys,
> >
> > Errors happen, that is normal. But I wonder if we need some clearer standards for testing downported patches? Currently the only requirement is to run tier1 tests, but nothing is mentioned about the configuration and the platforms to be tested. 11u is a stable maintenance release and in theory it should have a higher, not a lower bar for testing than the head release.
>
> I totally agree. This was my fault as I was lulled to a false sense of
> safety by the GH actions which I completely forgot were not happening
> for 11u.
>
> >
> > Surely testing just release is not enough, the debug builds need to be tested too. Then, arguably one should test on more than one platform. It's fine for individual contributors which don't have the resources, but larger companies should be able to test 11u patches on multiple platforms. Otherwise the burden of running the full gamut of tests and analysing accumulated errors lies overly by the 11u maintainers.
>
> IMO, a requirement to have a full spectrum of OS-HW combinations
> covered before pushing a backport will basically mean that the
> contributors without access to such resources will not be able/allowed
> to push backports at all.
> I am not saying it is wrong - just flagging the possible result.

I agree it shouldn't be a requirement to have a full spectrum, but I
think it should be a requirement to have a wide spectrum, which is
what Thomas is suggesting as I understand.  Since we may start using
the GH actions, adding x86 Linux, Windows and OSX in both standard and
debug configurations doesn't sound implausible, less mainstream
platforms may let pass without a specific test unless the
maintainers/reviewers have concerns, in which case they should help
with the testing too.

The author may just say "sorry, I don't have a Windows box, can
someone test there?", and that should not be a blocker if no one
follows up. Overall, it's always very dynamic and very case-by-case,
but the automation on some specific widely available platforms helps a
lot and we should implement it.

Cheers,
Mario

-- 
Mario Torre
Manager, Software Engineering
Red Hat GmbH <https://www.redhat.com>
9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898