RFR: 8316461: Fix: make test outputs TEST SUCCESS after unsuccessful exit

[Magnus Ihse Bursie]

On Tue, 19 Sep 2023 14:44:26 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

> `make test` outputs `TEST SUCCESS` after unsuccessful exit of JVM. One example is bad VM flags.
> 
> I now have a reproducer for something that has puzzled me before (and most of my co-workers I think):
> 
> // first test uses -XX:+UseG1GC second and third uses -XX:+UseG1GCC (sic)
> `make clean run-test TEST=open/test/hotspot/jtreg/gc/arguments/TestCompressedClassFlags.java JTREG='JAVA_OPTIONS=-XX:+UseG1GC'` -> OK
> `make run-test TEST=open/test/hotspot/jtreg/gc/arguments/TestCompressedClassFlags.java JTREG='JAVA_OPTIONS=-XX:+UseG1GCC'` -> OK
> `rm -rf test-results/jtreg_open_test_hotspot_jtreg_gc_arguments_TestCompressedClassFlags_java`
> `make run-test TEST=open/test/hotspot/jtreg/gc/arguments/TestCompressedClassFlags.java JTREG='JAVA_OPTIONS=-XX:+UseG1GCC'` -> FAIL
> 
> The problem is that the success of the first test run seems to be cached and the second test-run seems to reuse the result of the first run.
> 
> This seems to fix the problem:
> 
> diff --git a/make/RunTests.gmk b/make/RunTests.gmk
> index 25dcdbb083d..489c3f839f8 100644
> --- a/make/RunTests.gmk
> +++ b/make/RunTests.gmk
> @@ -864,6 +864,7 @@ define SetupRunJtregTestBody
> 
>    clean-workdir-$1:
>         $$(RM) -r $$($1_TEST_SUPPORT_DIR)
> + $$(RM) -r $$($1_TEST_RESULTS_DIR)
> 
>    $1_COMMAND_LINE := \
>        $$(JAVA) $$($1_JTREG_LAUNCHER_OPTIONS) \
> ``` 
> I have started tier 1-5 testing, but they are not completed yet. I would also like to know if this change cooperates nicely with RETRY_COUNT and REPEAT_COUNT. To me, it seems they both should be able to be more or less stateless, i.e. they ought to be able to quit on failure or success without a mutating directory, but It would be good if I could get that confirmed (because I have a hard time understanding the make files).

This is really just an issue with jtreg testing, but the fix will apply to all kind of tests. 

However, I don't think this is so much of a problem. If you rerun a test, the realistic assumption is that the test result directory will contain results from the last run. If we clear the report directory before running, we will make sure that it will only contain data from the latest run. The only possible drawback is that there is a window in which you will lose the old data before you got any new data, but that would only happen in case of a failure aborting the test run, and in that case, there is still the risk of the files in the directory being mixed from two runs, which does not seem good either.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/15820#issuecomment-1725892725