JDK 9 RFR of JDK-8078334: Mark regression tests using randomness

[Joseph D. Darcy]

Hello,

I'd like to get this changeset, or something close to it, pushed soon so 
we can start taking advantage of better failure triaging.

Any further concerns?

Thanks,

-Joe

On 4/24/2015 11:04 AM, joe darcy wrote:
> On 4/23/2015 10:58 PM, Alan Bateman wrote:
>> On 24/04/2015 02:54, Joseph D. Darcy wrote:
>>> Hello,
>>>
>>> Any additional comments on marking with tests in question with a 
>>> "randomness" keyword?
>>>
>>> Thanks,
>>>
>>> -Joe
>>>
>> I don't object to this keyword although I should say that most of the 
>> (apparent) randomness that I've seen hasn't been because of tests 
>> using Random but rather because of non-deterministic sequence of 
>> tests running in the same VM. The switch from othervm to agentvm a 
>> few years ago took a long time to weed out issues, same thing (even 
>> more) with -concurrency. Another source of apparent randomness is 
>> just having a large pool of machines and where machines are randomly 
>> selected to execute the tests.
>>
>> I agree with your comments that tests using Random should have report 
>> the seed and a have a way to re-run with the same value. It could 
>> help with some cases.
>>
>> One thing that isn't clear to me is how this keyword will be 
>> maintained. If I modify a test and replace the use of Random then I 
>> guess I need to remember to also remove this label. When I add a test 
>> that uses Random ...   Just wondering if there is a tool to do the 
>> tests analysis, identify the use of specific APIs ,and help identify 
>> where the keyword might be missing or not needed.
>>
>
> The goal of the keyword is to enable better analysis of bug failures. 
> We (still) have some intermittent test failures in the platform, some 
> probably remaining from -concurrency issues and some due to 
> inconsistent machine config issues on the test farms. However, there 
> is reason to believe some of the tests also fail because of the use of 
> randomness in the generation of test cases. For example, the test
>
>     java/lang/invoke/MethodHandles/CatchExceptionTest.java
>
> fails about once every three hundred runs when the random cases it 
> generates trigger an error condition. Setting the seed to known-bad 
> value causes the test to fail consistently (JDK-8055269). I have 
> suspicions some of our other intermittently failing tests are in a 
> similar but undiagnosed situation.
>
> If a test uses randomness, I think the default assumed cause for a 
> test failure should switch from environmental causes (bad machine 
> config, etc.) to the random behavior. Therefore, when a random-using 
> test fails, I think the failure analysis needs to include recording of 
> the bad seed to see if the failure is reproducible / deterministic 
> with a fixed seed value.
>
> I was very surprised by the large number of random-using tests in the 
> JDK regression tests. Nearly all of these have been stably passing for 
> many years. After the one-time cost of this large update to add the 
> keywords, there would be a small amount of incremental test 
> maintenance as new random-using tests were added or the randomness 
> value of a test changed. However, I would expect whether or not a test 
> used randomness to be a property of a test that very rarely changed.
>
> Thanks,
>
> -Joe