Clarification regarding the GitHub Actions pre-submit testing
shade at redhat.com
Fri Mar 19 09:02:47 UTC 2021
On 3/18/21 11:03 PM, Stuart Marks wrote:
> Well, maybe we should be clear on what you mean by a litmus test. When I read it, I
> thought, "GHA must be green before integration." If that's what you mean, then I
> would disagree with this policy. (And I note Magnus did as well.)
And here I thought I successfully avoided the word "must" to avoid this impression. D'oh.
To me more precise, I would say "GHA should be green before integration", where "should" is as per
ISO definition: "should" indicates a recommendation, where "a recommendation is defined as an
"expression [...] conveying a suggested possible choice or course of action deemed to be
particularly suitable without necessarily mentioning or excluding others."
I keep drawing parallels with jdk-submit, because I believe GHA is the spiritual successor for
*) Like jdk-submit, GHA is not a requirement, but a strong recommendation; Reviewers can ask for
jdk-submit/GHA runs, and contributors are encouraged to be proactive and run jdk-submit/GHA without
Reviewers explicitly asking;
*) Like jdk-submit, GHA can be skipped if you are reasonably sure your testing is comprehensive
(i.e. you run through vendor test systems); still, the strong recommendation to run jdk-submit/GHA
*) Like with jdk-submit, GHA alone is not always sufficient to cover all testing bases;
*) Like with jdk-submit, the failures with GHA can be reasonably ignored;
In my original reply, I said "...like with jdk-submit before, you can reasonably ignore some
failures, by rationally arguing they are not related to the current PR." .
After yesterday's discussion, that can be amended to "You can reasonably ignore GHA failures by
securing the reviewers agreement that: a) failures are not related to the current PR; or b)
failures, while related, are beyond the scope of this PR and/or contributor capabilities, and the
breakage is, while unfortunate, intentional, known and recorded in these follow-up bugs".
I think that is a reasonable thing to do, process-wise: failures do not necessarily block
integration, simple failures are detected and hopefully fixed before integration, complex failures
that require followups become known right away.
(Off to enjoy my day off; to be continued on Monday...)
More information about the jdk-dev