RFR: 8316461: Fix: make test outputs TEST SUCCESS after unsuccessful exit [v2]

[Erik Joelsson]

On Wed, 20 Sep 2023 07:49:20 GMT, Leo Korinth <lkorinth at openjdk.org> wrote:

>> `make test` outputs `TEST SUCCESS` after unsuccessful exit of JVM. One example is bad VM flags.
>> 
>> I now have a reproducer for something that has puzzled me before (and most of my co-workers I think):
>> 
>> // first test uses -XX:+UseG1GC second and third uses -XX:+UseG1GCC (sic)
>> `make clean run-test TEST=open/test/hotspot/jtreg/gc/arguments/TestCompressedClassFlags.java JTREG='JAVA_OPTIONS=-XX:+UseG1GC'` -> OK
>> `make run-test TEST=open/test/hotspot/jtreg/gc/arguments/TestCompressedClassFlags.java JTREG='JAVA_OPTIONS=-XX:+UseG1GCC'` -> OK
>> `rm -rf test-results/jtreg_open_test_hotspot_jtreg_gc_arguments_TestCompressedClassFlags_java`
>> `make run-test TEST=open/test/hotspot/jtreg/gc/arguments/TestCompressedClassFlags.java JTREG='JAVA_OPTIONS=-XX:+UseG1GCC'` -> FAIL
>> 
>> The problem is that the success of the first test run seems to be cached and the second test-run seems to reuse the result of the first run.
>> 
>> This seems to fix the problem:
>> 
>> diff --git a/make/RunTests.gmk b/make/RunTests.gmk
>> index 25dcdbb083d..489c3f839f8 100644
>> --- a/make/RunTests.gmk
>> +++ b/make/RunTests.gmk
>> @@ -864,6 +864,7 @@ define SetupRunJtregTestBody
>> 
>>    clean-workdir-$1:
>>         $$(RM) -r $$($1_TEST_SUPPORT_DIR)
>> + $$(RM) -r $$($1_TEST_RESULTS_DIR)
>> 
>>    $1_COMMAND_LINE := \
>>        $$(JAVA) $$($1_JTREG_LAUNCHER_OPTIONS) \
>> ``` 
>> I have started tier 1-5 testing, but they are not completed yet. I would also like to know if this change cooperates nicely with RETRY_COUNT and REPEAT_COUNT. To me, it seems they both should be able to be more or less stateless, i.e. they ought to be able to quit on failure or success without a mutating directory, but It would be good if I could get that confirmed (because I have a hard time understanding the make files).
>
> Leo Korinth has updated the pull request incrementally with one additional commit since the last revision:
> 
>   clean-workdir-$1 -> clean-outputdirs-$1

Marked as reviewed by erikj (Reviewer).

-------------

PR Review: https://git.openjdk.org/jdk/pull/15820#pullrequestreview-1635551535