Test policy follow-up, third testing tier

[Martijn Verburg]

Hi Joe,

Just a thought from the outside looking in (feel free to discard!).  I'm
seeing a reasonably strong trend to label these types of test suites/tiers
as: Unit, Integration, System and Heisen.  It's a very small thing, but
perhaps OpenJDK as a project could use some of this (hopefully familiar)
terminology.  e.g. I know that if I was a new developer joining OpenJDK
that if Tier 2 was re-labelled Heisen then I'd immediately know what that
meant.

Apart from that, I really like the proposal, I know it's been personally
frustrating as a casual contributor to see a Heisen test fail when making a
fairly trivial change.

As an aside the Adoption Group has an unofficial nightly build of jtreg and
some other code tools at:
https://adopt-openjdk.ci.cloudbees.com/view/OpenJDK%20code-tools/

Cheers,
Martijn

On 23 June 2015 at 23:48, joe darcy <joe.darcy at oracle.com> wrote:

> Hello,
>
> As a reminder, JDK 9 is now using a tiered testing policy where about
> 9,000 tests have been placed into one of two tiers, with tier 1 tests
> having a stricter policy on passing than tier 2. [1] The overall policy is
> that tier 1 tests should always pass in master and that integrations, both
> dev -> master and hotspot main -> dev, should preserve the property that
> all tier 1 tests pass. In addition, only a limited number of tier 2 tests
> should fail.
>
> In related work, jtreg keywords are now used to mark tests as using
> randomness and/or being known to fail intermittently. A test using
> randomness which is also known to fail intermittently should be modified so
> that the random seed is recorded in the test log and so that the seed can
> be set to see if the test reproducibly fails with a particular seed value.
> [2] Analysis of a test failure of a test using randomness should include
> recording the seed value in a bug.
>
> There are a number of expectations for regression tests in the JDK. There
> is a general expectation that a test passes reliably and quickly across
> platforms. Tests specific to particular platform can indicate that using
> the jtreg @requires feature. [3] Tests must pass with assertions and system
> assertions enabled, as controlled by the jtreg options -ea and -esa,
> respectively. In addition, tests should pass when invoked under jtreg's
> agent vm mode with concurrency, jtreg options -agentvm and -conc:$NUMBER.
> If a test for technical reasons cannot be run in agentvm mode, it should be
> written to explicitly run in othervm mode (@run main/othervm TestName) or
> use other equivalent measures, such as including the directory hosting the
> test on the othervm.dirs property in TEST.ROOT. That way such a test will
> still pass when invoked by a jtreg command which otherwise uses agentvm
> mode.
>
> Developers should use a current version of jtreg for test runs as the test
> suite is relying on features in new jtreg versions. [4] The jdk repo is
> declared to require at least jtreg 4.1 b11; jtreg 4.1 b12 is the latest
> version as of writing this message. One or more additional jtreg version
> can be expected before JDK 9 ships.
>
> The next iteration of refining tiered testing is to add a third tier, tier
> 3. The initial tier 1 and tier 2 test definitions did not include any
> client library tests. As mentioned as a possibility in the earlier tiered
> testing proposal, tier 3 can start hosting client library tests, beginning
> with sets of tests which do not require a "headful" environment. [5]
> Detailed discussions about which test sets to include will be conducted
> with the appropriate client libs teams. Additionally, for areas with known
> intermittent test problems, tests can be moved down to a lower test tier to
> ease getting clean results on the higher tier. For example, despite many
> improvements made over the last several years to the speed and robustness
> of the rmi tests, the rmi tests are still prone to intermittent failures
> when run concurrently. Therefore, as follow-up work after tier 3 is added,
> I propose moving the rmi tests from tier 2 to tier 3. [6]
>
> Comments?
>
> Thanks,
>
> -Joe
>
> [1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html
>
> [2] http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-April/002164.html
>
> [3] http://openjdk.java.net/jtreg/tag-spec.html#requires_names
>
> [4] For download information, see http://openjdk.java.net/jtreg/
>
> [5] 8081547: Prepare client libs regression tests for running in a
> concurrent, headless jtreg environment
>
> [6] JDK-8129624: Move jdk_rmi test group from tier 2 to tier 3
>