[External] : Re: Clarification regarding the GitHub Actions pre-submit testing

[Stuart Marks]

On 3/19/21 2:02 AM, Aleksey Shipilev wrote:
> And here I thought I successfully avoided the word "must" to avoid this impression. 
> D'oh.
> 
> To me more precise, I would say "GHA should be green before integration", where 
> "should" is as per ISO definition: "should" indicates a recommendation, where "a 
> recommendation is defined as an "expression [...] conveying a suggested possible 
> choice or course of action deemed to be particularly suitable without necessarily 
> mentioning or excluding others."
> ...
> After yesterday's discussion, that can be amended to "You can reasonably ignore GHA 
> failures by securing the reviewers agreement that: a) failures are not related to 
> the current PR; or b) failures, while related, are beyond the scope of this PR 
> and/or contributor capabilities, and the breakage is, while unfortunate, 
> intentional, known and recorded in these follow-up bugs".
> 
> I think that is a reasonable thing to do, process-wise: failures do not necessarily 
> block integration, simple failures are detected and hopefully fixed before 
> integration, complex failures that require followups become known right away.

OK, I'm glad to hear there's some pragmatism here. What you're saying is quite 
reasonable.

Your nuanced statement, while generally agreeable (at least to me) still allows a 
situation where disagreements over expectations and responsibilities can occur. I 
think it's fairly important to be explicit about these. The reason is that while I 
think we can generally assume that most people on here are reasonable, it's not 
valid to assume that reasonable people will all come to the same conclusion on 
everything.

What makes this complicated is that the GHA submit actions aren't just one thing. 
There's a whole bunch of stuff, with lots of different configuration options about 
what gets run on which platform. It's important to make sure that decisions about 
these configurations, and changes to them, are discussed and decided openly, and 
that adjustments to expectations and responsibilities are also communicated accordingly.

Let me take a hypothetical example of enabling tier2 tests in GHA. (Just to be 
clear, nobody has proposed this. This is something I just made up.) We might want to 
do this in order to improve test coverage and to provide earlier warning of actual 
failures, especially to people outside of Oracle. However, the tier2 tests include 
networking tests, which are notoriously dependent on machine configurations, 
timeouts, etc., and thus are susceptible to intermittent failures. A consequence of 
this change would then be that the GHA board would have red spots on it more of the 
time.

If there's open discussion of this, and everyone is clear that getting better test 
coverage is worth tolerating more red on the GHA board, then that's ok. If the 
decision is NOT to enable tier2, because keeping the GHA board green is more 
important than improving test coverage, that's ok too.

What I do NOT want to see is a situation where a few people get together and decide 
to make some change, and this is a surprise to the rest of the project. This leads 
to mismatches in expectations and responsibilities. I could see the following 
conversation ensue:

  - Hey, why am I suddenly getting red spots on the GHA board?
  - Those are intermittent failures from tier2.
  - When did we start running those?
  - The other day; I think there was a message on build-dev about it.
  - Well I can't integrate since the board isn't green; what am I supposed to do now?
  - Oh, just ignore the intermittent failures, they're ok.
  - When did that rule change?
  - There was no rule change, it's accommodated by the policy expressed in the
    middle of some email from Shipilev buried in the middle of a thread on jdk-dev
    last year....
  - etc.

The way to avoid such problems is to have clear communication about changes to the 
GHA submit actions, especially those that change responsibilities of all JDK 
developers. (This would be a good topic for a Committers' Workshop, but as we know, 
circumstances have prevented these from occurring.) Meanwhile, email to jdk-dev will 
have to suffice.

s'marks