RFR: 8253865: Pre-submit testing using GitHub Actions does not detect failures reliably
Robin Westberg
rwestberg at openjdk.java.net
Wed Sep 30 17:05:00 UTC 2020
The pre-submit test definitions utilizing GitHub Actions can sometimes report success even when there are failing
tests. The failure detection depended on make returning a non-zero exit code on failures, which doesn't seem to work.
To properly determine test outcome we should check the "test-summary.txt" result file for the string "TEST SUCCESS". If
it isn't found, the tests must have failed. This change also includes a reliability improvement.
Here's an example of a run with the updated test definitions: https://github.com/rwestberg/jdk/actions/runs/280529475
Note! There is a failure now properly detected in langtools/tier1 on Windows
(tools/javac/launcher/SourceLauncherTest.java) which is tracked by https://bugs.openjdk.java.net/browse/JDK-8249095 -
could be of interest to either fix or ProblemList that one before integrating this change.
-------------
Commit messages:
- Retry artifact downloading on failure
- Check results and show comprehensive list of failing tests (if any)
Changes: https://git.openjdk.java.net/jdk/pull/437/files
Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=437&range=00
Issue: https://bugs.openjdk.java.net/browse/JDK-8253865
Stats: 108 lines in 1 file changed: 108 ins; 0 del; 0 mod
Patch: https://git.openjdk.java.net/jdk/pull/437.diff
Fetch: git fetch https://git.openjdk.java.net/jdk pull/437/head:pull/437
PR: https://git.openjdk.java.net/jdk/pull/437
More information about the build-dev
mailing list