RFR: 8324659: GHA: Generic jtreg errors are not reported
Aleksey Shipilev
shade at openjdk.org
Thu Jan 25 14:42:49 UTC 2024
When looking at [JDK-8324647](https://bugs.openjdk.org/browse/JDK-8324647), I was surprised to see that GHA has this output:
Running test 'jtreg:test/lib-test:tier1'
/home/runner/work/jdk/jdk/test/lib-test/TEST.groups: group all: group includes itself
Error: One or more groups are invalid
Finished running test 'jtreg:test/lib-test:tier1'
Test report is stored in build/run-test-prebuilt/test-results/jtreg_test_lib_test_tier1
==============================
Test summary
==============================
TEST TOTAL PASS FAIL ERROR
>> jtreg:test/lib-test:tier1 1 0 0 1 <<
==============================
TEST FAILURE
...and yet, the tests are recorded as "green"! I think this is because our error reporting code is tad buggy. AFAICS, `make test-prebuilt` does not report a non-zero exit code on failure, and the external script looks for `build/run-test-prebuilt/make-support/exit-with-error` to check for error. But it does not _set_ the exit code on its own when the failure is discovered, neither it sets `failure=true` on generic failure.
I think at very minimum we should report `failure=true` on generic failure. I thought about reporting non-zero exit code from the parsing script, but it does not feel clean to return non-zero exit code from the _parsing script_. Non-zero exit code from that script should signify the error in the script itself.
Additional testing:
- [x] Current GHA with lib-test failure (now red)
- [x] Current GHA with lib-test fix (now green)
- [x] GHA with artificial errors: https://github.com/shipilev/jdk/actions/runs/7652797646/job/20854472816
-------------
Commit messages:
- Merge branch 'master' into JDK-83246590-gha-jtreg-failures
- Report just the failure=true
- Also set failure=true on generic error
- Fix
Changes: https://git.openjdk.org/jdk/pull/17568/files
Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=17568&range=00
Issue: https://bugs.openjdk.org/browse/JDK-8324659
Stats: 1 line in 1 file changed: 1 ins; 0 del; 0 mod
Patch: https://git.openjdk.org/jdk/pull/17568.diff
Fetch: git fetch https://git.openjdk.org/jdk.git pull/17568/head:pull/17568
PR: https://git.openjdk.org/jdk/pull/17568
More information about the build-dev
mailing list