RFR: 8024140: [TESTBUG] Profile based regression test groups for jdk repo

[David Holmes]

With thanks to Alan and Chris for their reviews I will push this initial 
set of group definitions.

David

On 3/09/2013 1:02 PM, David Holmes wrote:
> webrev:
>
> http://cr.openjdk.java.net/~dholmes/8024140/webrev/
>
> Similar to what was recently done on the hotspot side, this introduces
> profile-based test groups for the jdk regression tests.
>
> The primary groups are:
>   - jdk
>   - jre
>   - compact3
>   - compact2
>   - compact2_minimal
>   - compact1
>   - compact1_minimal
>
> The minimal VM is only supported on compact1 and compact2.
>
> The groups are defined hierarchically in two forms:
> - The need_xxx groups list all the tests that have a dependency on a
> specific profile. This is either because it tests a feature in that
> profile, or the test infrastructure uses a feature in that profile.
> - The primary groups are defined in terms of the other primary groups
> combined with the needs_xxx groups (including and excluding them as
> appropriate). For example the jre can run all tests from compact3, plus
> those from needs_jre, but excluding those from need_jdk.
>
> The bottom group defines all the actual tests to be considered, simply
> by listing the top-level test directories, and then excluding all the
> needs_xxx groups.
>
> To select a group of tests you use :<group-name>
>
> Eg to run only those tests that can run on compact1 use:
>
> jtreg :compact1
>
> Of course you still need to point jtreg at the right kind of runtime
> image (and give it a full JDK as the compile-jdk!); and if testing the
> minimal VM you need to tell jtreg to select it using -javaoptions:-minimal
>
> These top-level groups are not as useful standalone as they are for
> hotspot due to the fact that there are so many regression tests that are
> problematic in one way or another. In most cases what is desired is a
> way to run sub-set of tests for a given profile eg all jdk_core tests
> that can run on compact1. At present this would require a new group to
> be defined for each permutation and that is not scalable. I hope that in
> the near future jtreg will allow you to implicitly define such ad-hoc
> groups via the command-line, simply by listing groups to include and
> groups to exclude.
>
> The full jtreg group facility is only available in the most recent jtreg
> builds, so you will need to grab the latest nightly build, or latest
> sources.
>
> Note: once this is in place, anyone writing regression tests will need
> to be aware of whether that test is limited to certain profiles and
> update the group file accordingly. Sometimes it is not the item being
> tested that determines the minimum needed profile, but the test
> infrastructure eg if it uses XML.
>
> See TEST.groups for more information.
>
> Note: The initial group definitions proposed here are not complete.
> There are over 4000 tests of the "desktop" that can not be readily
> executed manually and it may be that a number of these tests will
> require a JDK rather than the full JRE. Our full test processes will
> discover this and update the lists as needed. (Or we fix the tests to
> not need full JDKs.)
>
> Thanks,
> David