RFR: 8324659: GHA: Generic jtreg errors are not reported

[Thomas Stuefe]

On Thu, 25 Jan 2024 12:02:35 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> When looking at [JDK-8324647](https://bugs.openjdk.org/browse/JDK-8324647), I was surprised to see that GHA has this output:
> 
> 
> Running test 'jtreg:test/lib-test:tier1'
> /home/runner/work/jdk/jdk/test/lib-test/TEST.groups: group all: group includes itself
> Error: One or more groups are invalid
> Finished running test 'jtreg:test/lib-test:tier1'
> Test report is stored in build/run-test-prebuilt/test-results/jtreg_test_lib_test_tier1
> 
> ==============================
> Test summary
> ==============================
>    TEST TOTAL PASS FAIL ERROR
>>> jtreg:test/lib-test:tier1 1 0 0 1 <<
> ==============================
> TEST FAILURE
> 
> 
> ...and yet, the tests are recorded as "green"! I think this is because our error reporting code is tad buggy. AFAICS, `make test-prebuilt` does not report a non-zero exit code on failure, and the external script looks for `build/run-test-prebuilt/make-support/exit-with-error` to check for error. But it does not _set_ the exit code on its own when the failure is discovered, neither it sets `failure=true` on generic failure.
> 
> I think at very minimum we should report `failure=true` on generic failure. I thought about reporting non-zero exit code from the parsing script, but it does not feel clean to return non-zero exit code from the _parsing script_. Non-zero exit code from that script should signify the error in the script itself.
> 
> Additional testing:
>  - [x] Current GHA with lib-test failure (now red)
>  - [x] Current GHA with lib-test fix (now green)
>  - [x] GHA with artificial errors: https://github.com/shipilev/jdk/actions/runs/7652797646/job/20854472816

Ouch. Wow. +1. 

Is this problem also in older releases?

-------------

Marked as reviewed by stuefe (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/17568#pullrequestreview-1845283123