Proposed new policies for JDK 9 regression tests: tiered testing, intermittent failures, and randomness

[Chris Hegarty]

On 19 Mar 2015, at 00:53, Brian Burkhalter <brian.burkhalter at oracle.com> wrote:

> 
> On Mar 18, 2015, at 5:43 PM, joe darcy <joe.darcy at oracle.com> wrote:
> 
>> Some tests want to explore a large space a inputs, often a space of inputs so large is is impractical or undesirable to exhaustively explore the space in routine testing. One way to get better test coverage in this kind of situation over time is for a test of the area to use randomness to explore a different subset of the input space of different runs. This use of randomness can be a valid testing technique, but it must be used with care.
>> 
>> If such a random-using test fails intermittently, it may be because it is encountering a real flaw in the product, but a flaw that is only visited rarely. If such a failure is observed, it should be added to the bug database along with the seed value that can be used to reproduce the situation. As a corollary, all such random-using tests should print out the seed value they are using a provide a way to set the seed value on a given run.
> 
> An example wherein this type of situation was recently improved may be found via this thread:
> 
> http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-March/032092.html

I like this solution, and have adopted something similar for a test that is currently in progress. We should, eventually, migrate this functionality to the test library.

>> To aid those analyzing the failure of a random-using test, a new jtreg keyword like "uses-randomness" should be added.
> 
> I concur that this would be helpful.

+1

-Chris.

> Brian