Test policy follow-up, third testing tier

[Alejandro E Murillo]

On 6/23/2015 4:48 PM, joe darcy wrote:
> Hello,
>
> As a reminder, JDK 9 is now using a tiered testing policy where about 
> 9,000 tests have been placed into one of two tiers, with tier 1 tests 
> having a stricter policy on passing than tier 2. [1] The overall 
> policy is that tier 1 tests should always pass in master and that 
> integrations, both dev -> master and hotspot main -> dev, should 
> preserve the property that all tier 1 tests pass. In addition, only a 
> limited number of tier 2 tests should fail.
Why is not everyone pushing to dev required to make sure tier1 is not 
broken?
Alejandro
>
> In related work, jtreg keywords are now used to mark tests as using 
> randomness and/or being known to fail intermittently. A test using 
> randomness which is also known to fail intermittently should be 
> modified so that the random seed is recorded in the test log and so 
> that the seed can be set to see if the test reproducibly fails with a 
> particular seed value. [2] Analysis of a test failure of a test using 
> randomness should include recording the seed value in a bug.
>
> There are a number of expectations for regression tests in the JDK. 
> There is a general expectation that a test passes reliably and quickly 
> across platforms. Tests specific to particular platform can indicate 
> that using the jtreg @requires feature. [3] Tests must pass with 
> assertions and system assertions enabled, as controlled by the jtreg 
> options -ea and -esa, respectively. In addition, tests should pass 
> when invoked under jtreg's agent vm mode with concurrency, jtreg 
> options -agentvm and -conc:$NUMBER. If a test for technical reasons 
> cannot be run in agentvm mode, it should be written to explicitly run 
> in othervm mode (@run main/othervm TestName) or use other equivalent 
> measures, such as including the directory hosting the test on the 
> othervm.dirs property in TEST.ROOT. That way such a test will still 
> pass when invoked by a jtreg command which otherwise uses agentvm mode.
>
> Developers should use a current version of jtreg for test runs as the 
> test suite is relying on features in new jtreg versions. [4] The jdk 
> repo is declared to require at least jtreg 4.1 b11; jtreg 4.1 b12 is 
> the latest version as of writing this message. One or more additional 
> jtreg version can be expected before JDK 9 ships.
>
> The next iteration of refining tiered testing is to add a third tier, 
> tier 3. The initial tier 1 and tier 2 test definitions did not include 
> any client library tests. As mentioned as a possibility in the earlier 
> tiered testing proposal, tier 3 can start hosting client library 
> tests, beginning with sets of tests which do not require a "headful" 
> environment. [5] Detailed discussions about which test sets to include 
> will be conducted with the appropriate client libs teams. 
> Additionally, for areas with known intermittent test problems, tests 
> can be moved down to a lower test tier to ease getting clean results 
> on the higher tier. For example, despite many improvements made over 
> the last several years to the speed and robustness of the rmi tests, 
> the rmi tests are still prone to intermittent failures when run 
> concurrently. Therefore, as follow-up work after tier 3 is added, I 
> propose moving the rmi tests from tier 2 to tier 3. [6]
>
> Comments?
>
> Thanks,
>
> -Joe
>
> [1] 
> http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html
>
> [2] 
> http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-April/002164.html
>
> [3] http://openjdk.java.net/jtreg/tag-spec.html#requires_names
>
> [4] For download information, see http://openjdk.java.net/jtreg/
>
> [5] 8081547: Prepare client libs regression tests for running in a 
> concurrent, headless jtreg environment
>
> [6] JDK-8129624: Move jdk_rmi test group from tier 2 to tier 3

-- 
Alejandro