Test policy follow-up, third testing tier

[joe darcy]

Hello,

As a reminder, JDK 9 is now using a tiered testing policy where about 
9,000 tests have been placed into one of two tiers, with tier 1 tests 
having a stricter policy on passing than tier 2. [1] The overall policy 
is that tier 1 tests should always pass in master and that integrations, 
both dev -> master and hotspot main -> dev, should preserve the property 
that all tier 1 tests pass. In addition, only a limited number of tier 2 
tests should fail.

In related work, jtreg keywords are now used to mark tests as using 
randomness and/or being known to fail intermittently. A test using 
randomness which is also known to fail intermittently should be modified 
so that the random seed is recorded in the test log and so that the seed 
can be set to see if the test reproducibly fails with a particular seed 
value. [2] Analysis of a test failure of a test using randomness should 
include recording the seed value in a bug.

There are a number of expectations for regression tests in the JDK. 
There is a general expectation that a test passes reliably and quickly 
across platforms. Tests specific to particular platform can indicate 
that using the jtreg @requires feature. [3] Tests must pass with 
assertions and system assertions enabled, as controlled by the jtreg 
options -ea and -esa, respectively. In addition, tests should pass when 
invoked under jtreg's agent vm mode with concurrency, jtreg options 
-agentvm and -conc:$NUMBER. If a test for technical reasons cannot be 
run in agentvm mode, it should be written to explicitly run in othervm 
mode (@run main/othervm TestName) or use other equivalent measures, such 
as including the directory hosting the test on the othervm.dirs property 
in TEST.ROOT. That way such a test will still pass when invoked by a 
jtreg command which otherwise uses agentvm mode.

Developers should use a current version of jtreg for test runs as the 
test suite is relying on features in new jtreg versions. [4] The jdk 
repo is declared to require at least jtreg 4.1 b11; jtreg 4.1 b12 is the 
latest version as of writing this message. One or more additional jtreg 
version can be expected before JDK 9 ships.

The next iteration of refining tiered testing is to add a third tier, 
tier 3. The initial tier 1 and tier 2 test definitions did not include 
any client library tests. As mentioned as a possibility in the earlier 
tiered testing proposal, tier 3 can start hosting client library tests, 
beginning with sets of tests which do not require a "headful" 
environment. [5] Detailed discussions about which test sets to include 
will be conducted with the appropriate client libs teams. Additionally, 
for areas with known intermittent test problems, tests can be moved down 
to a lower test tier to ease getting clean results on the higher tier. 
For example, despite many improvements made over the last several years 
to the speed and robustness of the rmi tests, the rmi tests are still 
prone to intermittent failures when run concurrently. Therefore, as 
follow-up work after tier 3 is added, I propose moving the rmi tests 
from tier 2 to tier 3. [6]

Comments?

Thanks,

-Joe

[1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html

[2] http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-April/002164.html

[3] http://openjdk.java.net/jtreg/tag-spec.html#requires_names

[4] For download information, see http://openjdk.java.net/jtreg/

[5] 8081547: Prepare client libs regression tests for running in a 
concurrent, headless jtreg environment

[6] JDK-8129624: Move jdk_rmi test group from tier 2 to tier 3