RFR: 8274083: Update testing docs to mention tiered testing [v4]

[Aleksey Shipilev]

On Thu, 23 Sep 2021 11:58:02 GMT, Alan Bateman <alanb at openjdk.org> wrote:

>> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Drop GH link
>
> doc/testing.html line 79:
> 
>> 77: <ul>
>> 78: <li><p><code>tier1</code>: This is the lowest test tier. Multiple developers run these tests every day. Normally, at least this tier should be clean before integration. Because of the widespread use, the tests in <code>tier1</code> are carefully selected and optimized to run fast, and to run in the most stable manner. The test failures in <code>tier1</code> are usually followed up on quickly, either with fixes, or adding relevant tests to problem list. GitHub Actions workflows, if enabled, run <code>tier1</code> tests.</p></li>
>> 79: <li><p><code>tier2</code>: This test group covers even more ground. These contain, among other things, tests that either run for too long to be at <code>tier1</code>, tests for less stable and/or experimental features, tests for less essential JDK components (for example, jaxp).</p></li>
> 
> I think this a bit unfair on tier2. Tests in tier2 need be run by anyone making changes in the security area, I/O, networking and many other core APIs. In general I think anyone changing code in the libraries area needs to look at TEST.groups to get some idea which tier or finer test group to run.
> 
> I'm not sure about words like "less stable" and "experimental features".  Do we have tests for hotspot experimental features in hotspot_tier2?

See new commit.

> Tests in tier2 need be run by anyone making changes in the security area, I/O, networking and many other core APIs. In general I think anyone changing code in the libraries area needs to look at TEST.groups to get some idea which tier or finer test group to run.

True. That is why the text says that component tests come first, and then you run first N tiers to capture anything else. If any component test is not in the component test group, and only in `tier2`, I'd say that is a test grouping bug, and the test should be in component test group to begin with. In other words, `tier2` testing does not replace the component group testing, and thus the requirement to run `tier2` if particular areas are changed would feel awkward to me.

>  Do we have tests for hotspot experimental features in hotspot_tier2?

We do move deprecating things out of `tier1` (see e.g. recent JDK-8272838).

-------------

PR: https://git.openjdk.java.net/jdk/pull/5615