RFR: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test

[Aleksey Shipilev]

On Tue, 4 Jan 2022 23:39:45 GMT, Sergey Bylokhov <serb at openjdk.org> wrote:

>> This adds the test repeat feature in the build system. This is convenient to follow-up on intermittently failing tests: the `REPEAT_COUNT > 0` would run the test multiple times, until we run out of repeats or the tests fail.
>> 
>> With this sample test:
>> 
>> 
>> /*
>>  * @test
>>  * @run driver IntermittentTest
>>  */
>> public class IntermittentTest {
>>     public static void main(String... args) {
>>         if (Math.random() < 0.05) {
>>             throw new IllegalStateException("Oops");
>>         }
>>     }
>> }
>> 
>> 
>> ...a lucky run looks like this:
>> 
>> 
>> $ CONF=linux-x86_64-server-fastdebug make run-test TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=5"
>> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
>> Running tests using JTREG control variable 'REPEAT_COUNT=5'
>> Test selection 'sanity/IntermittentTest.java', will run:
>> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>> 
>> Running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
>> 
>> Repeating Jtreg run: 1 out of 5
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>> 
>> Repeating Jtreg run: 2 out of 5
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>> 
>> Repeating Jtreg run: 3 out of 5
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>> 
>> Repeating Jtreg run: 4 out of 5
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>> 
>> Repeating Jtreg run: 5 out of 5
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
>> Test report is stored in ...
>> 
>> ==============================
>> Test summary
>> ==============================
>>    TEST                                              TOTAL  PASS  FAIL ERROR   
>>    jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>>                                                          1     1     0     0   
>> ==============================
>> TEST SUCCESS
>> 
>> 
>> ...and unlucky run looks like this:
>> 
>> 
>> $ CONF=linux-x86_64-server-fastdebug make run-test TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=50"
>> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
>> Running tests using JTREG control variable 'REPEAT_COUNT=50'
>> Test selection 'sanity/IntermittentTest.java', will run:
>> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>> ...
>> Repeating Jtreg run: 9 out of 50
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>> 
>> Repeating Jtreg run: 10 out of 50
>> --------------------------------------------------
>> TEST: sanity/IntermittentTest.java
>> TEST JDK: ...
>> 
>> ...
>> 
>> STDERR:
>> java.lang.IllegalStateException: Oops
>> 	at IntermittentTest.main(IntermittentTest.java:8)
>> 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
>> 	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
>> 	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
>> 	at java.base/java.lang.Thread.run(Thread.java:833)
>> 
>> JavaTest Message: Test threw exception: java.lang.IllegalStateException
>> JavaTest Message: shutting down test
>> 
>> 
>> TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.IllegalStateException: Oops
>> --------------------------------------------------
>> Test results: failed: 1
>> Report written to ...
>> Results written to ...
>> Error: Some tests failed or other problems occurred.
>> 
>> Failures detected, no more repeats.
>> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
>> Test report is stored in ...
>> 
>> ==============================
>> Test summary
>> ==============================
>>    TEST                                              TOTAL  PASS  FAIL ERROR   
>>    jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>>>>                                                       1     0     1     0 <<
>> ==============================
>> TEST FAILURE
>
> doc/testing.html line 198:
> 
>> 196: <p>Retry failed tests up to a set number of times. Defaults to 0.</p>
>> 197: <h4 id="repeat_count">REPEAT_COUNT</h4>
>> 198: <p>Repeat the tests for a set number of times. Defaults to 0.</p>
> 
> Don't we need to mention that the goal of the repeat_count is to run a test until it fails first (to reproduce intermittent issues)?

We can. Current docs copy-pasted the form of `RETRY_COUNT` above. Both should be updated with some mention for use cases. Let me do that...

-------------

PR: https://git.openjdk.java.net/jdk/pull/6720