RFR: 8274083: Update testing docs to mention tiered testing [v2]

[Aleksey Shipilev]

On Wed, 22 Sep 2021 02:05:35 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> I'll take another stab at it tomorrow.
>> 
>> The whole tiered testing thing encapsulates that some tests are more important to run (at least, first or regularly) than others. So we need to somehow capture that. Discussing what tests are actually running is not relevant, because `TEST.groups` is the source of truth there.
>> 
>> For a contributor, these descriptions should tell why, when and how often would you use a particular tier. In other words, good docs capture some thinking about how contributors are supposed to use the existing tests. It seems to be a broad consensus that we run `tier1` before non-trivial integrations. It seems to be a rule of thumb that we care a lot about `tier1` stability and speed, and we push out less stable, slower or specially-configured tests to higher tiers. As I inspect the test groups, I think `tier2` is for regular but non-`tier1`-grade tests, `tier3` is mostly for stress tests, and `tier4` is (somewhat by construction) is "everything else".
>> 
>> From here, this what I'd like to tell contributors: run `tier1` early and often, the bugs there are likely to be real product bugs and should be looked at as soon as possible, run `tier1` + `tier2` to execute all regular tests but suspect test bugs more, run `tier3` to add even more stress testing and suspect environmental problems when they fail, run `tier4` to run everything that previous tiers missed and hope for the best. Maybe I should just put that in the docs :)
>
> Ideal world: run every test on every platform before every integration.
> Real world: run as many tests as is practical having consideration for time of testing and the cost of testing resources, and the scope of the change.
> 
> It is assumed/expected you will have run all tests pertaining to the area of code change, that can reasonably be expected to be run.
> 
> The tiered testing is intended to give broader coverage to ensure no unintended consequences of a change. At a minimum run all tier1 tests on all non-trivial integrations. If you have time and resources, and think a change may have broader impact than what has been explicitly tested, then run further tiers.

All right, thanks for the inputs. Let me try again, see new commit.

-------------

PR: https://git.openjdk.java.net/jdk/pull/5615