Need reviewers - jdk testing changes 6888927

[Kelly O'Hair]

Martin Buchholz wrote:
> On Wed, Nov 4, 2009 at 09:19, Kelly O'Hair <Kelly.Ohair at sun.com> wrote:
> 
>> Managing the ProblemList (as Jon noted) will be a pain at times.
> 
> It's even more of a problem as development becomes more distributed,
> as failures become platform-dependent, etc...
> 
> I would prefer to see per-file annotations that jtreg can understand.

Not sure how annotating tests is any better in a distributed world,
or if tests failed in a platform-dependent way. Both seem nasty.

I have no objection and would actually prefer to see the testcases
annotated, with @ignore or whatever keywords people want to invent.
But I came to the conclusion that in general, relying on that as the sole
way to exclude unstable tests was just not good enough, and may not happen
for a long long time.
I would hope that eventually, this ProblemList shrinks, perhaps to
nothingness someday, I know I know,  ... dream on. :^(

The issue I saw was that regressions could sneak in all kinds of ways,
and at any given point in time, a testcase could become unstable
due to an OS patch, or a hotspot change, or something outside of the
jdk repository.

So we need a quick and easy way to exclude, and stop having dozens
of developers investigate the same known failures.
Yes, in many cases bugs should be filed, jdk bugs or testcase bugs,
but my first priority was to find that list of tests that pass reliably,
and stick my wizard's staff in the ground and say 'Thou shall pass!'. ;^)

Then see if we could start requiring a set of tests to always pass.

Next, I start going through the ProblemList in detail, with various
team help if possible, and see if we can shrink this problem down.

> 
> I also like the approach I advocated some years ago:
> - run tests under JDK 1
> - run tests under JDK 2
> - diff the results (JTreport directories)
> See my diff-javatest script.
> jtreg now has support for this (I haven't tried it lately).

So you want everyone to run the tests twice?
Why run 6,000+ tests when you could run 3,000+ tests?

Now running all the ProblemList tests this way might be interesting,
if some of them didn't crash jtreg. :^(

-kto

> 
> Martin