RFR: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test

[Aleksey Shipilev]

This adds the test repeat feature in the build system. This is convenient to follow-up on intermittently failing tests: the `REPEAT_COUNT > 0` would run the test multiple times, until we run out of repeats or the tests fail.

With this sample test:


/*
 * @test
 * @run driver IntermittentTest
 */
public class IntermittentTest {
    public static void main(String... args) {
        if (Math.random() < 0.05) {
            throw new IllegalStateException("Oops");
        }
    }
}


...a lucky run looks like this:


$ CONF=linux-x86_64-server-fastdebug make run-test TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=5"
Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
Running tests using JTREG control variable 'REPEAT_COUNT=5'
Test selection 'sanity/IntermittentTest.java', will run:
* jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java

Running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'

Repeating Jtreg run: 1 out of 5
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...

Repeating Jtreg run: 2 out of 5
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...

Repeating Jtreg run: 3 out of 5
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...

Repeating Jtreg run: 4 out of 5
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...

Repeating Jtreg run: 5 out of 5
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...
Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
Test report is stored in ...

==============================
Test summary
==============================
   TEST                                              TOTAL  PASS  FAIL ERROR   
   jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
                                                         1     1     0     0   
==============================
TEST SUCCESS


...and unlucky run looks like this:


$ CONF=linux-x86_64-server-fastdebug make run-test TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=50"
Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
Running tests using JTREG control variable 'REPEAT_COUNT=50'
Test selection 'sanity/IntermittentTest.java', will run:
* jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
...
Repeating Jtreg run: 9 out of 50
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...

Repeating Jtreg run: 10 out of 50
--------------------------------------------------
TEST: sanity/IntermittentTest.java
TEST JDK: ...

...

STDERR:
java.lang.IllegalStateException: Oops
	at IntermittentTest.main(IntermittentTest.java:8)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
	at java.base/java.lang.Thread.run(Thread.java:833)

JavaTest Message: Test threw exception: java.lang.IllegalStateException
JavaTest Message: shutting down test


TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.IllegalStateException: Oops
--------------------------------------------------
Test results: failed: 1
Report written to ...
Results written to ...
Error: Some tests failed or other problems occurred.

Failures detected, no more repeats.
Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
Test report is stored in ...

==============================
Test summary
==============================
   TEST                                              TOTAL  PASS  FAIL ERROR   
   jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>>                                                       1     0     1     0 <<
==============================
TEST FAILURE

-------------

Commit messages:
 - Fix

Changes: https://git.openjdk.java.net/jdk/pull/6720/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=6720&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8244602
  Stats: 28 lines in 3 files changed: 27 ins; 0 del; 1 mod
  Patch: https://git.openjdk.java.net/jdk/pull/6720.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/6720/head:pull/6720

PR: https://git.openjdk.java.net/jdk/pull/6720