RFR: 8253865: Pre-submit testing using GitHub Actions does not detect failures reliably

Robin Westberg rwestberg at openjdk.java.net
Wed Sep 30 17:05:00 UTC 2020


The pre-submit test definitions utilizing GitHub Actions can sometimes report success even when there are failing
tests. The failure detection depended on make returning a non-zero exit code on failures, which doesn't seem to work.

To properly determine test outcome we should check the "test-summary.txt" result file for the string "TEST SUCCESS". If
it isn't found, the tests must have failed. This change also includes a reliability improvement.

Here's an example of a run with the updated test definitions: https://github.com/rwestberg/jdk/actions/runs/280529475

Note! There is a failure now properly detected in langtools/tier1 on Windows
(tools/javac/launcher/SourceLauncherTest.java) which is tracked by https://bugs.openjdk.java.net/browse/JDK-8249095 -
could be of interest to either fix or ProblemList that one before integrating this change.

-------------

Commit messages:
 - Retry artifact downloading on failure
 - Check results and show comprehensive list of failing tests (if any)

Changes: https://git.openjdk.java.net/jdk/pull/437/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=437&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8253865
  Stats: 108 lines in 1 file changed: 108 ins; 0 del; 0 mod
  Patch: https://git.openjdk.java.net/jdk/pull/437.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/437/head:pull/437

PR: https://git.openjdk.java.net/jdk/pull/437



More information about the build-dev mailing list