Proposed new policies for JDK 9 regression tests: tiered testing, intermittent failures, and randomness

[Brian Burkhalter]

On Mar 18, 2015, at 5:43 PM, joe darcy <joe.darcy at oracle.com> wrote:

> Some tests want to explore a large space a inputs, often a space of inputs so large is is impractical or undesirable to exhaustively explore the space in routine testing. One way to get better test coverage in this kind of situation over time is for a test of the area to use randomness to explore a different subset of the input space of different runs. This use of randomness can be a valid testing technique, but it must be used with care.
> 
> If such a random-using test fails intermittently, it may be because it is encountering a real flaw in the product, but a flaw that is only visited rarely. If such a failure is observed, it should be added to the bug database along with the seed value that can be used to reproduce the situation. As a corollary, all such random-using tests should print out the seed value they are using a provide a way to set the seed value on a given run.

An example wherein this type of situation was recently improved may be found via this thread:

http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-March/032092.html

> To aid those analyzing the failure of a random-using test, a new jtreg keyword like "uses-randomness" should be added.

I concur that this would be helpful.

Brian