Need reviewers - jdk testing changes 6888927

[Kelly O'Hair]

Martin Buchholz wrote:
> One of the things you've done is to create "test sets".
> I did something like that in my "jtr" script (ask Tim if you can't find it).
> For each logical component, it's non-trivial to find all the tests
> for that (e.g. how to test string handling). 
> This is especially true for shared directories like java/lang
> and java/util.  I don't think there's much value in creating a special
> target for testing java/util, since
> 1) jtreg can do that easily already
> 2) this doesn't match the job of any particular technology developer

With the jtreg options I set in the Makefile, there is a big
difference between just running 'jtreg java/util' vs. 'make jdk_util'.
The Makefile will exclude problem tests and @ignore tests, and
will use -samevm (which is not the jtreg default).
So I can somewhat guarantee that if all is good with the world,
'make jdk_util' will be 100% pass, and run as quickly as possible.

That's what started me on this road, running 'jtreg java/util' is
slow and unstable, and most of all, when it finishes I don't really
know if it was a good result or a bad result.

Don't mean to pick on java/util, the same is true with any of these
batches. I kept wondering, "I got N failures, is that good?" :^(

And with -samevm, it is valuable to run "test sets" as a way of
isolating the possible problems with tests sharing a VM.

> 
> What would be really useful is to try to fix all the tests
> that fail under -samevm (perhaps by marking them
> @run othervm)

I started down that road myself and fell into a big rat hole. :^(
Modifying tests requires a buy in from those teams, and just the
logistics of that effort was more than I had the time for.

Also, it's not clear that just making them run othervm is the
right thing to do in some cases. In talking to some developers,
some want to go the extra mile to make the test samevm safe,
others don't. Some don't want to, or have the time, to do anything
to their tests. So it's a large mix of attitudes and plans on how
to deal with it.

So the ProblemList was my answer, and a testcase checklist for teams
that want to fix their tests, and then delete the lines from ProblemList.

I'll probably try and fix some of the tests I'm familiar with in
the future, like the serviceability tests.

-kto

> 
> Martin
> 
> On Tue, Nov 3, 2009 at 18:03, Kelly O'Hair <Kelly.Ohair at sun.com 
> <mailto:Kelly.Ohair at sun.com>> wrote:
> 
> 
>     I need some formal reviewers on these jdk/test/Makefile changes.
> 
>     More polish and changes may need to be done later, but there
>     is value in what I have now, and I need lots of help to improve
>     things (and fix some of our testcases).
> 
>     Here is the bugid and webrev:
> 
>     6888927: Fix jdk jtreg tests to indicate which ones need othervm,
>     allow for use of samevm option
>     http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/
>     <http://cr.openjdk.java.net/%7Eohair/openjdk7/jdk7-build-samevm-6888927/webrev/>
> 
>     The goal was to see if we could easily run most of the regression
>     tests in the jdk/test directory, in minimum time, and without so
>     much noise (e.g. do not run unstable or problematic tests).
>     This could then be used as a benchmark to validate some quality measure
>     of the jdk7 that was built.
> 
>     The primary changes include:
>      * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>      * Addition of a jdk/test/ProblemList file to be used by
>     jdk/test/Makefile
> 
>     I tried to group all the tests (by directory names) into jdk_* sets,
>     then
>     tried to balance them as to how long each set ran and if they could
>     be run
>     with the jtreg -samevm option. The balancing was tricky, and will
>     need more
>     polish, along with the virtual frame buffer idea. I also had to give up
>     on the awt, rmi, and swing tests until the Xvfb issues are figured out.
> 
>     I chose to not actually modify the tests themselves if they needed
>     fixing,
>     it was just too overwhelming. So developers and teams may want to
>     browse the
>     ProblemList for their favorite tests (which aren't my favorites ;^) and
>     consider what they might want to do.
> 
>     With this new jdk/test/Makefile, anybody can:
> 
>      cd jdk/test
>      gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}]
>     [JT_HOME=${YOUR_JTREG_HOME}] jdk_all
> 
>     And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
>     machine isn't too slow and old. I was able to run it in 90 minutes on a
>     monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
> 
>     Then I added "-j 4" to the gnumake command line and it only took 30mins!
>     So the various jdk_* make targets can be run in parallel.
> 
>     Please let me know what you think.
> 
>     -kto
> 
> 
>