RFR: 8024140: [TESTBUG] Profile based regression test groups for jdk repo

[David Holmes]

On 3/09/2013 7:22 PM, Chris Hegarty wrote:
> David,
>
> I share Alan's concerns about the maintainability of this, but I don't
> have a better answer ( without much reorganizing/tagging ).
>
> Out of interest, how did you come up with the lists? For example,
> java/net/URLClassLoader/closetest/GetResourceAsStream.java is in
> needs_jdk. I presume because of a dependency on javac. The test could be
> trivially changed to remove this dependency ( I don't expect you to have
> the time to do this for all tests ). Maybe, over time, with some
> socializing of these find of issues, enough tests can be changed/moved
> to greatly reduce the number of specific tests explicitly listed in
> these groups.

I came up with the lists by starting with a division based around the 
profiles API and then running all the tests. I then had to examine all 
the failures and determine why they failed. Fortunately the bulk of 
these fell into two camps:
- NoClassDefFoundError
- exit code 1

The former are self-identifying with regards to the API dependency. The 
latter invariably meant a tool (javac, jps, jcmd etc) was being executed 
using the test-jdk.

It was a laborious process.

I think most of the incidental needs_jdk tests really just need to be 
changed to use tools from the compile-jdk. But that is certainly an 
issue for the component teams.

> I resolved some of the ftp issues Alan referred to, a while back, and I
> guess this helped reduce the need to have a few more specific tests
> listed in needs_jdk. This really seems like a task for the actual
> component teams, with higher level guidance.
>
> Once this gets in I'll take a look to see if it is possible to remove
> some of the obvious networking tests. What is the recommended test cycle
> to verify that all this is working well after making changes?

That's part of the problem here. We don't build profiles as part of the 
promotion builds so there are no binaries for anyone to simply point to. 
I did my own profile builds, and added the minimal VM, to be able to 
test things. Using "javac -profile xxx" can be of use in determining an 
API dependency.

Thanks,
David

> -Chris.
>
> On 09/03/2013 04:02 AM, David Holmes wrote:
>> webrev:
>>
>> http://cr.openjdk.java.net/~dholmes/8024140/webrev/
>>
>> Similar to what was recently done on the hotspot side, this introduces
>> profile-based test groups for the jdk regression tests.
>>
>> The primary groups are:
>>   - jdk
>>   - jre
>>   - compact3
>>   - compact2
>>   - compact2_minimal
>>   - compact1
>>   - compact1_minimal
>>
>> The minimal VM is only supported on compact1 and compact2.
>>
>> The groups are defined hierarchically in two forms:
>> - The need_xxx groups list all the tests that have a dependency on a
>> specific profile. This is either because it tests a feature in that
>> profile, or the test infrastructure uses a feature in that profile.
>> - The primary groups are defined in terms of the other primary groups
>> combined with the needs_xxx groups (including and excluding them as
>> appropriate). For example the jre can run all tests from compact3, plus
>> those from needs_jre, but excluding those from need_jdk.
>>
>> The bottom group defines all the actual tests to be considered, simply
>> by listing the top-level test directories, and then excluding all the
>> needs_xxx groups.
>>
>> To select a group of tests you use :<group-name>
>>
>> Eg to run only those tests that can run on compact1 use:
>>
>> jtreg :compact1
>>
>> Of course you still need to point jtreg at the right kind of runtime
>> image (and give it a full JDK as the compile-jdk!); and if testing the
>> minimal VM you need to tell jtreg to select it using
>> -javaoptions:-minimal
>>
>> These top-level groups are not as useful standalone as they are for
>> hotspot due to the fact that there are so many regression tests that are
>> problematic in one way or another. In most cases what is desired is a
>> way to run sub-set of tests for a given profile eg all jdk_core tests
>> that can run on compact1. At present this would require a new group to
>> be defined for each permutation and that is not scalable. I hope that in
>> the near future jtreg will allow you to implicitly define such ad-hoc
>> groups via the command-line, simply by listing groups to include and
>> groups to exclude.
>>
>> The full jtreg group facility is only available in the most recent jtreg
>> builds, so you will need to grab the latest nightly build, or latest
>> sources.
>>
>> Note: once this is in place, anyone writing regression tests will need
>> to be aware of whether that test is limited to certain profiles and
>> update the group file accordingly. Sometimes it is not the item being
>> tested that determines the minimum needed profile, but the test
>> infrastructure eg if it uses XML.
>>
>> See TEST.groups for more information.
>>
>> Note: The initial group definitions proposed here are not complete.
>> There are over 4000 tests of the "desktop" that can not be readily
>> executed manually and it may be that a number of these tests will
>> require a JDK rather than the full JRE. Our full test processes will
>> discover this and update the lists as needed. (Or we fix the tests to
>> not need full JDKs.)
>>
>> Thanks,
>> David