RFR: 8274083: Update testing docs to mention tiered testing [v4]

[Alan Bateman]

On Thu, 23 Sep 2021 11:30:20 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions, let's mention them in `testing.md`. 
>> 
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Drop GH link

doc/testing.html line 72:

> 70: <p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p>
> 71: <h3 id="common-test-groups">Common Test Groups</h3>
> 72: <p>In the ideal world, contributors would be advised to run all the tests for every change. But in real world, one could only be expected to run as many tests as practical, while being mindful of the scope for the change, the testing resources available, etc.</p>

A suggestion here is to start with something like "Ideally, all tests are run for every change but this may not be practical. Contributors are expected to at least run tier1 (see below) and the tests for the areas that are changed."

doc/testing.html line 79:

> 77: <ul>
> 78: <li><p><code>tier1</code>: This is the lowest test tier. Multiple developers run these tests every day. Normally, at least this tier should be clean before integration. Because of the widespread use, the tests in <code>tier1</code> are carefully selected and optimized to run fast, and to run in the most stable manner. The test failures in <code>tier1</code> are usually followed up on quickly, either with fixes, or adding relevant tests to problem list. GitHub Actions workflows, if enabled, run <code>tier1</code> tests.</p></li>
> 79: <li><p><code>tier2</code>: This test group covers even more ground. These contain, among other things, tests that either run for too long to be at <code>tier1</code>, tests for less stable and/or experimental features, tests for less essential JDK components (for example, jaxp).</p></li>

I think this a bit unfair on tier2. Tests in tier2 need be run by anyone making changes in the security area, I/O, networking and many other core APIs. In general I think anyone changing code in the libraries area needs to look at TEST.groups to get some idea which tier or finer test group to run.

I'm not sure about words like "less stable" and "experimental features".  Do we have tests for hotspot experimental features in hotspot_tier2?

-------------

PR: https://git.openjdk.java.net/jdk/pull/5615