Clarification regarding the GitHub Actions pre-submit testing

[Aleksey Shipilev]

On 3/18/21 11:03 PM, Stuart Marks wrote:
> Well, maybe we should be clear on what you mean by a litmus test. When I read it, I
> thought, "GHA must be green before integration." If that's what you mean, then I
> would disagree with this policy. (And I note Magnus did as well.)

And here I thought I successfully avoided the word "must" to avoid this impression. D'oh.

To me more precise, I would say "GHA should be green before integration", where "should" is as per 
ISO definition: "should" indicates a recommendation, where "a recommendation is defined as an 
"expression [...] conveying a suggested possible choice or course of action deemed to be 
particularly suitable without necessarily mentioning or excluding others."

I keep drawing parallels with jdk-submit, because I believe GHA is the spiritual successor for 
jdk-submit:
  *) Like jdk-submit, GHA is not a requirement, but a strong recommendation; Reviewers can ask for 
jdk-submit/GHA runs, and contributors are encouraged to be proactive and run jdk-submit/GHA without 
  Reviewers explicitly asking;
  *) Like jdk-submit, GHA can be skipped if you are reasonably sure your testing is comprehensive 
(i.e. you run through vendor test systems); still, the strong recommendation to run jdk-submit/GHA 
still applies;
  *) Like with jdk-submit, GHA alone is not always sufficient to cover all testing bases;
  *) Like with jdk-submit, the failures with GHA can be reasonably ignored;

In my original reply, I said "...like with jdk-submit before, you can reasonably ignore some 
failures, by rationally arguing they are not related to the current PR." [1].

After yesterday's discussion, that can be amended to "You can reasonably ignore GHA failures by 
securing the reviewers agreement that: a) failures are not related to the current PR; or b) 
failures, while related, are beyond the scope of this PR and/or contributor capabilities, and the 
breakage is, while unfortunate, intentional, known and recorded in these follow-up bugs".

I think that is a reasonable thing to do, process-wise: failures do not necessarily block 
integration, simple failures are detected and hopefully fixed before integration, complex failures 
that require followups become known right away.

(Off to enjoy my day off; to be continued on Monday...)

-- 
Thanks,
-Aleksey

[1] https://mail.openjdk.java.net/pipermail/jdk-dev/2021-March/005194.html