JDK 9 RFR of JDK-8078334: Mark regression tests using randomness

[Joseph D. Darcy]

Hello,

Any additional comments on marking with tests in question with a 
"randomness" keyword?

Thanks,

-Joe

On 4/21/2015 7:13 PM, Joseph D. Darcy wrote:
> Hello,
>
> As the next step of implementing "Proposed new policies for JDK 9 
> regression tests: tiered testing, intermittent failures, and 
> randomness" [1], please review this changeset which adds a 
> "randomness" jtreg keyword and tags 240 regression tests in the jdk 
> repository with the keyword:
>
>     JDK-8078334: Mark regression tests using randomness
>     http://cr.openjdk.java.net/~darcy/8078334.0/
>
> The update to test/TEST.ROOT is
>
> -# The list of keywords supported in the entire test suite
> -keys=2d dnd i18n intermittent
> +# The list of keywords supported in the entire test suite.  The
> +# "intermittent" keyword marks tests known to fail intermittently.
> +# The "randomness" keyword marks tests using randomness with test
> +# cases differing from run to run. (A test using a fixed random seed
> +# would not count as "randomness" by this definition.) Extra care
> +# should be taken to handle test failures of intermittent or
> +# randomness tests.
> +
> +keys=2d dnd i18n intermittent randomness
>
> To find the affected files for this fix, I grepped over the regression 
> tests looking for "random" and verified if a random number generated 
> was used in a way where its values would differ from run to run. In 
> particular, as noted above, a java.util.Random random number generator 
> with a fixed seed did *not* qualify a test for the "randomness" keyword.
>
> The usual update to the files in this changeset is just
>
> + * @key randomness
>
> The updated tests are mostly in the core libs and security libs areas 
> with a few serviceability tests too. The list in this bug is not quite 
> exhaustive; I did not include rmi tests or client libraries tests.
>
> I verified that running
>
>     jtreg ... -l -k:randomness .
>
> returns the expected number of tests and that the tests continue to 
> pass (or be ignored) when run.
>
> One goal of marking the tests using randomness is to help root out 
> some remaining intermittent test failures. If one of the randomness 
> tests is observed to fail intermittently, if it has not already been 
> updated to print out the random seed and be able to accept a 
> particular seed when run, the test should be so modified so that 
> future failures can log the seed value in hopes of reproducing the 
> failure.
>
> Brian Burkhalter has proactively made these sorts of changes to the 
> java.math tests earlier in JDK 9. [2] Since most of the randomness 
> tests don't have a history of instability (and due to the unexpected 
> large number of them!), I don't think it is necessary to proactively 
> update all the other tests without evidence of a history of failure. 
> However, when an otherwise unexplained failure is seen, I recommend 
> applying the sort of updates Brian has done in java.math to the 
> failing test(s).
>
> Thanks,
>
> -Joe
>
> [1] 
> http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html
>
> [2] JDK-8074460, Always print seeds used in [Splittable]Random 
> instances in java.math tests,
> https://bugs.openjdk.java.net/browse/JDK-8074460