Need reviewers - jdk testing changes 6888927

[Kelly O'Hair]

Alan Bateman wrote:
> Kelly O'Hair wrote:
>>
>> I need some formal reviewers on these jdk/test/Makefile changes.
>>
>> More polish and changes may need to be done later, but there
>> is value in what I have now, and I need lots of help to improve
>> things (and fix some of our testcases).
>>
>> Here is the bugid and webrev:
>>
>> 6888927: Fix jdk jtreg tests to indicate which ones need othervm, 
>> allow for use of samevm option
>> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/ 
>>
>>
>> The goal was to see if we could easily run most of the regression
>> tests in the jdk/test directory, in minimum time, and without so
>> much noise (e.g. do not run unstable or problematic tests).
>> This could then be used as a benchmark to validate some quality measure
>> of the jdk7 that was built.
>>
>> The primary changes include:
>>   * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>>   * Addition of a jdk/test/ProblemList file to be used by 
>> jdk/test/Makefile
>>
>> I tried to group all the tests (by directory names) into jdk_* sets, then
>> tried to balance them as to how long each set ran and if they could be 
>> run
>> with the jtreg -samevm option. The balancing was tricky, and will need 
>> more
>> polish, along with the virtual frame buffer idea. I also had to give up
>> on the awt, rmi, and swing tests until the Xvfb issues are figured out.
>>
>> I chose to not actually modify the tests themselves if they needed 
>> fixing,
>> it was just too overwhelming. So developers and teams may want to 
>> browse the
>> ProblemList for their favorite tests (which aren't my favorites ;^) and
>> consider what they might want to do.
>>
>> With this new jdk/test/Makefile, anybody can:
>>
>>   cd jdk/test
>>   gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}] 
>> [JT_HOME=${YOUR_JTREG_HOME}] jdk_all
>>
>> And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
>> machine isn't too slow and old. I was able to run it in 90 minutes on a
>> monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
>>
>> Then I added "-j 4" to the gnumake command line and it only took 30mins!
>> So the various jdk_* make targets can be run in parallel.
>>
>> Please let me know what you think.
>>
>> -kto
>>
> This looks really good. I don't see any obvious issues in the Makefile 
> and I grabbed the patch to try it and it worked great.
> 
> My only concern is that ProblemList will require ongoing maintenance. I 
> realize you don't want to get into changing tests and adding @ignore to 
> unstable tests but I think you'll need to send a broadcast mail to make 
> sure that folks know that their tests have problems in samevm mode, that 
> they need to remove from ProblemList when fixing tests, maybe they need 
> to add to ProblemList if changing/adding tests that take a long time, etc.

Yeah, but I didn't feel like I had many good solutions to this issue.

I'll maintain it to some degree, but I need the people that use this
makefile to help me out, and for everyone to start looking at why
their tests are even on this list.

> 
> For the groupings, then the only odd one I noticed is that jdk_io 
> includes the javax/imagio tests. It would best to separate these as they 
> are very different areas.

It was more of a balancing act on the test groupings with regards to timings
etc., and imageio did have "io" in it's name :^), and most of them ran safely
in samevm mode. So we can move them to any other group that is run in samevm
mode.  Do you have a better match than jdk_io?


-kto

> 
> -Alan.