Need reviewers - jdk testing changes 6888927

[Kelly O'Hair]

Andrew John Hughes wrote:
> 2009/11/4 Kelly O'Hair <Kelly.Ohair at sun.com>:
>> I need some formal reviewers on these jdk/test/Makefile changes.
>>
>> More polish and changes may need to be done later, but there
>> is value in what I have now, and I need lots of help to improve
>> things (and fix some of our testcases).
>>
>> Here is the bugid and webrev:
>>
>> 6888927: Fix jdk jtreg tests to indicate which ones need othervm, allow for
>> use of samevm option
>> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/
>>
>> The goal was to see if we could easily run most of the regression
>> tests in the jdk/test directory, in minimum time, and without so
>> much noise (e.g. do not run unstable or problematic tests).
>> This could then be used as a benchmark to validate some quality measure
>> of the jdk7 that was built.
>>
>> The primary changes include:
>>  * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>>  * Addition of a jdk/test/ProblemList file to be used by jdk/test/Makefile
>>
>> I tried to group all the tests (by directory names) into jdk_* sets, then
>> tried to balance them as to how long each set ran and if they could be run
>> with the jtreg -samevm option. The balancing was tricky, and will need more
>> polish, along with the virtual frame buffer idea. I also had to give up
>> on the awt, rmi, and swing tests until the Xvfb issues are figured out.
>>
>> I chose to not actually modify the tests themselves if they needed fixing,
>> it was just too overwhelming. So developers and teams may want to browse the
>> ProblemList for their favorite tests (which aren't my favorites ;^) and
>> consider what they might want to do.
>>
>> With this new jdk/test/Makefile, anybody can:
>>
>>  cd jdk/test
>>  gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}] [JT_HOME=${YOUR_JTREG_HOME}]
>> jdk_all
>>
>> And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
>> machine isn't too slow and old. I was able to run it in 90 minutes on a
>> monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
>>
>> Then I added "-j 4" to the gnumake command line and it only took 30mins!
>> So the various jdk_* make targets can be run in parallel.
>>
>> Please let me know what you think.
>>
>> -kto
>>
>>
>>
> 
> I like this idea, the time to run a jdk jtreg test has put me off in the past.
> BTW, jtreg is the one thing I wouldn't mind having in the tree as it's
> not something people are likely to have installed.  IcedTea has a cut
> down copy and that means tests get run.
> 

I could add some makefile rules that detect when JT_HOME is not defined
or cannot be found, then do a wget of the jtreg bundle and installs it
in the build/*/ area for use. Just hate to have that repeatedly done
over and over. I usually keep the latest jtreg in my own ~/import area
along with the 1.7.1 ant and latest findbugs.

Not sure what the best answer is for insuring the latest jtreg is
available to use.

> I'm worried about the use of the IcedTea7 test list.  These were added
> just because they were the ones that failed under my testing and
> that's a while ago.  Just as may happen with your ProblemList, this
> hasn't been maintained recently so I'd be wary about including the
> list verbatim.
> 

Fair enough. I'll be excluding the IcedTea items that I added from the IcedTea
list for now, which were mostly awt and swing tests anyway, which
are not part of my jdk_all target yet due to the DISPLAY issues.
If any of them fail for me, I'll add them to the list, with a little
more detail as to where they fail. So this list will become an OpenJDK
list. At some point I want to review the list and make sure all the
appropriate bug reports are filed.

> With this patch in, IcedTea7 can just use the list provided here.  We
> don't use test/Makefile at present at all, and it would be nice to do
> so.  I need to look at the issues with that.

Well, I'm not sure you can if I removed the ones you excluded, but then
again, as stated, jdk_all does not include awt or swing tests yet.

If there are tests run by jdk_all that cause problems, we can add
them.

-kto