JDK 9 RFR of JDK-8078334: Mark regression tests using randomness

[joe darcy]

On 4/23/2015 10:58 PM, Alan Bateman wrote:
> On 24/04/2015 02:54, Joseph D. Darcy wrote:
>> Hello,
>>
>> Any additional comments on marking with tests in question with a 
>> "randomness" keyword?
>>
>> Thanks,
>>
>> -Joe
>>
> I don't object to this keyword although I should say that most of the 
> (apparent) randomness that I've seen hasn't been because of tests 
> using Random but rather because of non-deterministic sequence of tests 
> running in the same VM. The switch from othervm to agentvm a few years 
> ago took a long time to weed out issues, same thing (even more) with 
> -concurrency. Another source of apparent randomness is just having a 
> large pool of machines and where machines are randomly selected to 
> execute the tests.
>
> I agree with your comments that tests using Random should have report 
> the seed and a have a way to re-run with the same value. It could help 
> with some cases.
>
> One thing that isn't clear to me is how this keyword will be 
> maintained. If I modify a test and replace the use of Random then I 
> guess I need to remember to also remove this label. When I add a test 
> that uses Random ...   Just wondering if there is a tool to do the 
> tests analysis, identify the use of specific APIs ,and help identify 
> where the keyword might be missing or not needed.
>

The goal of the keyword is to enable better analysis of bug failures. We 
(still) have some intermittent test failures in the platform, some 
probably remaining from -concurrency issues and some due to inconsistent 
machine config issues on the test farms. However, there is reason to 
believe some of the tests also fail because of the use of randomness in 
the generation of test cases. For example, the test

     java/lang/invoke/MethodHandles/CatchExceptionTest.java

fails about once every three hundred runs when the random cases it 
generates trigger an error condition. Setting the seed to known-bad 
value causes the test to fail consistently (JDK-8055269). I have 
suspicions some of our other intermittently failing tests are in a 
similar but undiagnosed situation.

If a test uses randomness, I think the default assumed cause for a test 
failure should switch from environmental causes (bad machine config, 
etc.) to the random behavior. Therefore, when a random-using test fails, 
I think the failure analysis needs to include recording of the bad seed 
to see if the failure is reproducible / deterministic with a fixed seed 
value.

I was very surprised by the large number of random-using tests in the 
JDK regression tests. Nearly all of these have been stably passing for 
many years. After the one-time cost of this large update to add the 
keywords, there would be a small amount of incremental test maintenance 
as new random-using tests were added or the randomness value of a test 
changed. However, I would expect whether or not a test used randomness 
to be a property of a test that very rarely changed.

Thanks,

-Joe