RFR: 8024140: [TESTBUG] Profile based regression test groups for jdk repo

[David Holmes]

Hi Alan,

On 3/09/2013 5:55 PM, Alan Bateman wrote:
> On 03/09/2013 04:02, David Holmes wrote:
>> webrev:
>>
>> http://cr.openjdk.java.net/~dholmes/8024140/webrev/
>>
>> :
>>
>> Note: once this is in place, anyone writing regression tests will need
>> to be aware of whether that test is limited to certain profiles and
>> update the group file accordingly. Sometimes it is not the item being
>> tested that determines the minimum needed profile, but the test
>> infrastructure eg if it uses XML.
>>
> I don't think this is very maintainable and I don't think we should
> force everyone that adds/changes tests to have to also maintain this
> file. In general I worry that this approach will just not scale when we
> move to modules. That said, I know you are under short term pressure to
> put in a solution to allow the profiles be tested.

As I've said elsewhere, but for the public record I will repeat, 
whatever mechanism we use anyone writing regression tests has to start 
thinking about what dependencies their test has. Then they have to 
record those dependencies somewhere. While tagging a single test in the 
future seems fine, it is not fine to have to tag a few thousand tests 
initially. Conversely having to potentially edit one line in the groups 
file hardly seems burdensome.

But perhaps down the track we can use the group definitions as input to 
a tool that will auto-tag the tests - if/when a complete tagging scheme 
is devised.

As for the move to modules ... I think that is a whole new ball game.

> So my view is that we should start looking for a replacement to this as
> soon as we can. In the mean-time then I think we have to assume that it
> will get out of sync periodically and maybe it's possible to provide
> tooling (maybe based on jdeps) that would help keep it up to date.

Tooling/automation would of course be good, I just don't see the pieces 
in place as yet.

> As regards the definitions then one observation is the TestNG tests that
> you've listed in needs_compact2 are actually compact1 tests. I wonder if
> this is a TestNG dependency on XML rather than dependencies in the test
> themselves.

Yes. As noted in the file the dependencies are not only what is being 
tested but what test infrastructure is being used. The testng tests uses 
xml which restricts them to compact2. Some tests use jps/jcmd or 
tools.jar which restricts them to a full jdk.

> You've listed most of the tests in sun/nio/cs in needs_charsets and I
> wonder whether it would be simpler to just move these to sun/nio/cs/ext.
> We've already moved several tests (the ftp tests for example) in
> preparation for profiles and maybe this is an option here.

That is up to the nio-dev folk but I certainly wouldn't object! I was 
contemplating simply adding sun/nio/cs to needs_charsets, which would 
artificially restrict some of those tests, but simplify the expression. 
There are 67 tests, of which 41 actually need charsets.jar.

> I see that needs_jre includes jdk_jdi but JDI is not in the JRE. Is it
> possible that you are testing JDI in the compile JDK here?

Thanks! - that's a simple oversight on my part. Fixed.

Thanks for the review.

David

> -Alan.
>
>
>