RFR: 8024140: [TESTBUG] Profile based regression test groups for jdk repo

[Chris Hegarty]

David,

I share Alan's concerns about the maintainability of this, but I don't 
have a better answer ( without much reorganizing/tagging ).

Out of interest, how did you come up with the lists? For example, 
java/net/URLClassLoader/closetest/GetResourceAsStream.java is in 
needs_jdk. I presume because of a dependency on javac. The test could be 
trivially changed to remove this dependency ( I don't expect you to have 
the time to do this for all tests ). Maybe, over time, with some 
socializing of these find of issues, enough tests can be changed/moved 
to greatly reduce the number of specific tests explicitly listed in 
these groups.

I resolved some of the ftp issues Alan referred to, a while back, and I 
guess this helped reduce the need to have a few more specific tests 
listed in needs_jdk. This really seems like a task for the actual 
component teams, with higher level guidance.

Once this gets in I'll take a look to see if it is possible to remove 
some of the obvious networking tests. What is the recommended test cycle 
to verify that all this is working well after making changes?

-Chris.

On 09/03/2013 04:02 AM, David Holmes wrote:
> webrev:
>
> http://cr.openjdk.java.net/~dholmes/8024140/webrev/
>
> Similar to what was recently done on the hotspot side, this introduces
> profile-based test groups for the jdk regression tests.
>
> The primary groups are:
>   - jdk
>   - jre
>   - compact3
>   - compact2
>   - compact2_minimal
>   - compact1
>   - compact1_minimal
>
> The minimal VM is only supported on compact1 and compact2.
>
> The groups are defined hierarchically in two forms:
> - The need_xxx groups list all the tests that have a dependency on a
> specific profile. This is either because it tests a feature in that
> profile, or the test infrastructure uses a feature in that profile.
> - The primary groups are defined in terms of the other primary groups
> combined with the needs_xxx groups (including and excluding them as
> appropriate). For example the jre can run all tests from compact3, plus
> those from needs_jre, but excluding those from need_jdk.
>
> The bottom group defines all the actual tests to be considered, simply
> by listing the top-level test directories, and then excluding all the
> needs_xxx groups.
>
> To select a group of tests you use :<group-name>
>
> Eg to run only those tests that can run on compact1 use:
>
> jtreg :compact1
>
> Of course you still need to point jtreg at the right kind of runtime
> image (and give it a full JDK as the compile-jdk!); and if testing the
> minimal VM you need to tell jtreg to select it using -javaoptions:-minimal
>
> These top-level groups are not as useful standalone as they are for
> hotspot due to the fact that there are so many regression tests that are
> problematic in one way or another. In most cases what is desired is a
> way to run sub-set of tests for a given profile eg all jdk_core tests
> that can run on compact1. At present this would require a new group to
> be defined for each permutation and that is not scalable. I hope that in
> the near future jtreg will allow you to implicitly define such ad-hoc
> groups via the command-line, simply by listing groups to include and
> groups to exclude.
>
> The full jtreg group facility is only available in the most recent jtreg
> builds, so you will need to grab the latest nightly build, or latest
> sources.
>
> Note: once this is in place, anyone writing regression tests will need
> to be aware of whether that test is limited to certain profiles and
> update the group file accordingly. Sometimes it is not the item being
> tested that determines the minimum needed profile, but the test
> infrastructure eg if it uses XML.
>
> See TEST.groups for more information.
>
> Note: The initial group definitions proposed here are not complete.
> There are over 4000 tests of the "desktop" that can not be readily
> executed manually and it may be that a number of these tests will
> require a JDK rather than the full JRE. Our full test processes will
> discover this and update the lists as needed. (Or we fix the tests to
> not need full JDKs.)
>
> Thanks,
> David