11u testing (was: Re: [11u] RFR: 8258414: OldObjectSample events too expensive)

[Thomas Stüfe]

On Fri, Apr 23, 2021 at 4:56 PM Mario Torre <neugens at redhat.com> wrote:

> On Fri, Apr 23, 2021 at 4:33 PM Jaroslav Bachorík
> <jaroslav.bachorik at datadoghq.com> wrote:
> >
> > Hi,
> >
> > On Thu, Apr 22, 2021 at 7:19 AM Thomas Stüfe <thomas.stuefe at gmail.com>
> wrote:
> > >
> > > Hi guys,
> > >
> > > Errors happen, that is normal. But I wonder if we need some clearer
> standards for testing downported patches? Currently the only requirement is
> to run tier1 tests, but nothing is mentioned about the configuration and
> the platforms to be tested. 11u is a stable maintenance release and in
> theory it should have a higher, not a lower bar for testing than the head
> release.
> >
> > I totally agree. This was my fault as I was lulled to a false sense of
> > safety by the GH actions which I completely forgot were not happening
> > for 11u.
> >
> > >
> > > Surely testing just release is not enough, the debug builds need to be
> tested too. Then, arguably one should test on more than one platform. It's
> fine for individual contributors which don't have the resources, but larger
> companies should be able to test 11u patches on multiple platforms.
> Otherwise the burden of running the full gamut of tests and analysing
> accumulated errors lies overly by the 11u maintainers.
> >
> > IMO, a requirement to have a full spectrum of OS-HW combinations
> > covered before pushing a backport will basically mean that the
> > contributors without access to such resources will not be able/allowed
> > to push backports at all.
> > I am not saying it is wrong - just flagging the possible result.
>
> I agree it shouldn't be a requirement to have a full spectrum, but I
> think it should be a requirement to have a wide spectrum, which is
> what Thomas is suggesting as I understand.  Since we may start using
> the GH actions, adding x86 Linux, Windows and OSX in both standard and
> debug configurations doesn't sound implausible, less mainstream
> platforms may let pass without a specific test unless the
> maintainers/reviewers have concerns, in which case they should help
> with the testing too.
>
> The author may just say "sorry, I don't have a Windows box, can
> someone test there?", and that should not be a blocker if no one
> follows up.


Why not? In your hypothetical case probably reviewers are overwhelmed and
no-one has found time to build and test for the contributor. IMHO that
should not be an automatic okay-to-push. If it's not time critical,
it's better if the contributor waits then to push an untested fix.

Where more exotic hardware is concerned (e.g. ppc) I agree, but even then -
if the patch explicitly touches architecture specific stuff, it should be
tested on that architecture before pushing.


> Overall, it's always very dynamic and very case-by-case,
> but the automation on some specific widely available platforms helps a
> lot and we should implement it.
>

I agree.


>
> Cheers,

Mario
>
> --
> Mario Torre
> Manager, Software Engineering
> Red Hat GmbH <https://www.redhat.com>
> 9704 A60C B4BE A8B8 0F30  9205 5D7E 4952 3F65 7898
>
>