RFR: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test

[Erik Joelsson]

On Mon, 6 Dec 2021 11:12:22 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> This adds the test repeat feature in the build system. This is convenient to follow-up on intermittently failing tests: the `REPEAT_COUNT > 0` would run the test multiple times, until we run out of repeats or the tests fail.
> 
> With this sample test:
> 
> 
> /*
>  * @test
>  * @run driver IntermittentTest
>  */
> public class IntermittentTest {
>     public static void main(String... args) {
>         if (Math.random() < 0.05) {
>             throw new IllegalStateException("Oops");
>         }
>     }
> }
> 
> 
> ...a lucky run looks like this:
> 
> 
> $ CONF=linux-x86_64-server-fastdebug make run-test TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=5"
> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
> Running tests using JTREG control variable 'REPEAT_COUNT=5'
> Test selection 'sanity/IntermittentTest.java', will run:
> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
> 
> Running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> 
> Repeating Jtreg run: 1 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
> 
> Repeating Jtreg run: 2 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
> 
> Repeating Jtreg run: 3 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
> 
> Repeating Jtreg run: 4 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
> 
> Repeating Jtreg run: 5 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> Test report is stored in ...
> 
> ==============================
> Test summary
> ==============================
>    TEST                                              TOTAL  PASS  FAIL ERROR   
>    jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>                                                          1     1     0     0   
> ==============================
> TEST SUCCESS
> 
> 
> ...and unlucky run looks like this:
> 
> 
> $ CONF=linux-x86_64-server-fastdebug make run-test TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=50"
> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
> Running tests using JTREG control variable 'REPEAT_COUNT=50'
> Test selection 'sanity/IntermittentTest.java', will run:
> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
> ...
> Repeating Jtreg run: 9 out of 50
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
> 
> Repeating Jtreg run: 10 out of 50
> --------------------------------------------------
> TEST: sanity/IntermittentTest.java
> TEST JDK: ...
> 
> ...
> 
> STDERR:
> java.lang.IllegalStateException: Oops
> 	at IntermittentTest.main(IntermittentTest.java:8)
> 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
> 	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
> 	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
> 	at java.base/java.lang.Thread.run(Thread.java:833)
> 
> JavaTest Message: Test threw exception: java.lang.IllegalStateException
> JavaTest Message: shutting down test
> 
> 
> TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.IllegalStateException: Oops
> --------------------------------------------------
> Test results: failed: 1
> Report written to ...
> Results written to ...
> Error: Some tests failed or other problems occurred.
> 
> Failures detected, no more repeats.
> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> Test report is stored in ...
> 
> ==============================
> Test summary
> ==============================
>    TEST                                              TOTAL  PASS  FAIL ERROR   
>    jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>>>                                                       1     0     1     0 <<
> ==============================
> TEST FAILURE

Marked as reviewed by erikj (Reviewer).

-------------

PR: https://git.openjdk.java.net/jdk/pull/6720