Test policy follow-up, third testing tier

[joe darcy]

Hi Martijn,

On 6/24/2015 3:22 AM, Martijn Verburg wrote:
> Hi Joe,
>
> Just a thought from the outside looking in (feel free to discard!).  
> I'm seeing a reasonably strong trend to label these types of test 
> suites/tiers as: Unit, Integration, System and Heisen.  It's a very 
> small thing, but perhaps OpenJDK as a project could use some of this 
> (hopefully familiar) terminology.  e.g. I know that if I was a new 
> developer joining OpenJDK that if Tier 2 was re-labelled Heisen then 
> I'd immediately know what that meant.

Thanks for the feedback on naming. The tiers as they are currently 
constituted don't align well with the divisions you're outlined. 
However, I think more informative naming for the tiers is something to 
keep in mind to revisit in the future.

Cheers,

-Joe

>
> Apart from that, I really like the proposal, I know it's been 
> personally frustrating as a casual contributor to see a Heisen test 
> fail when making a fairly trivial change.
>
> As an aside the Adoption Group has an unofficial nightly build of 
> jtreg and some other code tools at: 
> https://adopt-openjdk.ci.cloudbees.com/view/OpenJDK%20code-tools/
>
> Cheers,
> Martijn
>
> On 23 June 2015 at 23:48, joe darcy <joe.darcy at oracle.com 
> <mailto:joe.darcy at oracle.com>> wrote:
>
>     Hello,
>
>     As a reminder, JDK 9 is now using a tiered testing policy where
>     about 9,000 tests have been placed into one of two tiers, with
>     tier 1 tests having a stricter policy on passing than tier 2. [1]
>     The overall policy is that tier 1 tests should always pass in
>     master and that integrations, both dev -> master and hotspot main
>     -> dev, should preserve the property that all tier 1 tests pass.
>     In addition, only a limited number of tier 2 tests should fail.
>
>     In related work, jtreg keywords are now used to mark tests as
>     using randomness and/or being known to fail intermittently. A test
>     using randomness which is also known to fail intermittently should
>     be modified so that the random seed is recorded in the test log
>     and so that the seed can be set to see if the test reproducibly
>     fails with a particular seed value. [2] Analysis of a test failure
>     of a test using randomness should include recording the seed value
>     in a bug.
>
>     There are a number of expectations for regression tests in the
>     JDK. There is a general expectation that a test passes reliably
>     and quickly across platforms. Tests specific to particular
>     platform can indicate that using the jtreg @requires feature. [3]
>     Tests must pass with assertions and system assertions enabled, as
>     controlled by the jtreg options -ea and -esa, respectively. In
>     addition, tests should pass when invoked under jtreg's agent vm
>     mode with concurrency, jtreg options -agentvm and -conc:$NUMBER.
>     If a test for technical reasons cannot be run in agentvm mode, it
>     should be written to explicitly run in othervm mode (@run
>     main/othervm TestName) or use other equivalent measures, such as
>     including the directory hosting the test on the othervm.dirs
>     property in TEST.ROOT. That way such a test will still pass when
>     invoked by a jtreg command which otherwise uses agentvm mode.
>
>     Developers should use a current version of jtreg for test runs as
>     the test suite is relying on features in new jtreg versions. [4]
>     The jdk repo is declared to require at least jtreg 4.1 b11; jtreg
>     4.1 b12 is the latest version as of writing this message. One or
>     more additional jtreg version can be expected before JDK 9 ships.
>
>     The next iteration of refining tiered testing is to add a third
>     tier, tier 3. The initial tier 1 and tier 2 test definitions did
>     not include any client library tests. As mentioned as a
>     possibility in the earlier tiered testing proposal, tier 3 can
>     start hosting client library tests, beginning with sets of tests
>     which do not require a "headful" environment. [5] Detailed
>     discussions about which test sets to include will be conducted
>     with the appropriate client libs teams. Additionally, for areas
>     with known intermittent test problems, tests can be moved down to
>     a lower test tier to ease getting clean results on the higher
>     tier. For example, despite many improvements made over the last
>     several years to the speed and robustness of the rmi tests, the
>     rmi tests are still prone to intermittent failures when run
>     concurrently. Therefore, as follow-up work after tier 3 is added,
>     I propose moving the rmi tests from tier 2 to tier 3. [6]
>
>     Comments?
>
>     Thanks,
>
>     -Joe
>
>     [1]
>     http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html
>
>     [2]
>     http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-April/002164.html
>
>     [3] http://openjdk.java.net/jtreg/tag-spec.html#requires_names
>
>     [4] For download information, see http://openjdk.java.net/jtreg/
>
>     [5] 8081547: Prepare client libs regression tests for running in a
>     concurrent, headless jtreg environment
>
>     [6] JDK-8129624: Move jdk_rmi test group from tier 2 to tier 3
>
>