Clarification regarding the GitHub Actions pre-submit testing

[Volker Simonis]

This is a very good discussion which is important to have. I want to
share my thoughts as a long-term contributor to this project (my first
changeset is from 2008) and long-term maintainer of what was called
_Secondary_ports_ like linux-{ppc64,s390} and aix-ppc64:

- First of all, I think that the current GH Actions testing is by far
the *best* system we ever had! It's not only worth supporting it but
instead we should do everything possible to improve it.
- It is also the *most democratic* system we every had because it
removes the difference between "internal" (i.e. Oracle employees) and
"external" (i.e. everybody else) Contributors which I never liked .
This distinction is obviously not an explicit part of any of the
OpenJDK project rules or bylaws but has factually existed for a long
time (e.g. in the beginning only "internal" Contributors could push
HotSpot changes and later only "internal" Contributors could see the
full results of submit-repo tests).
- From my experience, ALL Contribuors ("internal" and "external") have
ALWAYS been supportive and willing to help port maintainers to
maintain their ports in the event of breaking changes. The problem has
never been the good will and intent on the side of Contributors but
rather the absence of good tools and infrastructure.
- The hurdles for a port to be accepted into the OpenJDK mainline are
quite high and if ports are not used or supported any more they get
removed quickly (e.g. Solaris, SPARC). I'd call such ports "official"
JDK ports in contrast to ports which exist outside the OpenJDK project
or outside the main repository. As long as "official" ports can be
easily be tested/verified by GH Actions I think it is reasonable and
feasible to keep them at least buildable. For official OpenJDK ports
without GH Actions support this remains the responsibility of the
maintainers. It might be nice to have a kind of automatic notification
process for such ports in the future (e.g. special commit commands
like "/breaks-AIX"?).
- I think there's a little confusion in the previous post whether new
features need to support ALL "official" ports or if they should just
not break them. While I agree that the former requirement is excessive
and unrealistic, the latter should be a must, at least to the extent
that's easily verifiable by the automated GH Action testing. For small
changes such a requirement is usually trivially achievable. For larger
features like Loom or Valhalla the effort is a little higher but still
reasonable. Such features are developed for years and bringing the
final version into a shape which can easily by stubbed out on
unsupported platforms is not an undue request but merely a matter of
proper design. It's also not unreasonable for such features which has
been developed for several years to wait a few more weeks until the
build works on ALL "official" OpenJDK platforms.
- There were concerns about the  reliability and availability of GH
Actions. In my opinion these concerns are either invalid or they apply
just as well to the whole GitHub infrastructure we depend on. If we
trust the latter, why not the former as well? It was also mentioned
that the current GH Actions infrastructure is only maintained by
"volunteers" and that's why we can't rely on it. To me this sounds a
little like "internal"/"external" Contributor problem mentioned
before. There's nothing that prevents skeptics from jumping in and
supporting these volunteers :)

The current system is already pretty good. Aleksey's definition of how
it should be used (see below) is to the point. Of coarse there's
always room for improvements but instead of criticizing the current
system I think we should rather focus on improving it.

Best regards,
Volker

On Fri, Mar 19, 2021 at 10:03 AM Aleksey Shipilev <shade at redhat.com> wrote:
>
> On 3/18/21 11:03 PM, Stuart Marks wrote:
> > Well, maybe we should be clear on what you mean by a litmus test. When I read it, I
> > thought, "GHA must be green before integration." If that's what you mean, then I
> > would disagree with this policy. (And I note Magnus did as well.)
>
> And here I thought I successfully avoided the word "must" to avoid this impression. D'oh.
>
> To me more precise, I would say "GHA should be green before integration", where "should" is as per
> ISO definition: "should" indicates a recommendation, where "a recommendation is defined as an
> "expression [...] conveying a suggested possible choice or course of action deemed to be
> particularly suitable without necessarily mentioning or excluding others."
>
> I keep drawing parallels with jdk-submit, because I believe GHA is the spiritual successor for
> jdk-submit:
>   *) Like jdk-submit, GHA is not a requirement, but a strong recommendation; Reviewers can ask for
> jdk-submit/GHA runs, and contributors are encouraged to be proactive and run jdk-submit/GHA without
>   Reviewers explicitly asking;
>   *) Like jdk-submit, GHA can be skipped if you are reasonably sure your testing is comprehensive
> (i.e. you run through vendor test systems); still, the strong recommendation to run jdk-submit/GHA
> still applies;
>   *) Like with jdk-submit, GHA alone is not always sufficient to cover all testing bases;
>   *) Like with jdk-submit, the failures with GHA can be reasonably ignored;
>
> In my original reply, I said "...like with jdk-submit before, you can reasonably ignore some
> failures, by rationally arguing they are not related to the current PR." [1].
>
> After yesterday's discussion, that can be amended to "You can reasonably ignore GHA failures by
> securing the reviewers agreement that: a) failures are not related to the current PR; or b)
> failures, while related, are beyond the scope of this PR and/or contributor capabilities, and the
> breakage is, while unfortunate, intentional, known and recorded in these follow-up bugs".
>
> I think that is a reasonable thing to do, process-wise: failures do not necessarily block
> integration, simple failures are detected and hopefully fixed before integration, complex failures
> that require followups become known right away.
>
> (Off to enjoy my day off; to be continued on Monday...)
>
> --
> Thanks,
> -Aleksey
>
> [1] https://mail.openjdk.java.net/pipermail/jdk-dev/2021-March/005194.html
>