How to test OpenJDK?

[Jonathan Gibbons]

There is no reason why it should not be possible to have 100% pass rate
for most of the jdk tests as well.   If there are problematic tests in  
the jdk
test suite, we should identify those tests, and then logically  
partition the
tests into two groups -- those that should always be expected to pass,  
and
those that don't, for whatever reason. That at least would allow to to  
set
a quality bar for any OpenJDK build of which tests should always pass.

Frankly, I've never understood the attitude that it is OK to have  
tests that
fail, and which don't deserve attention, but as long as we have such  
tests,
we should identify them, using any of the available jtreg mechanisms,
such as keywords, @ignore, or exclude lists.

And, all the other, unmarked tests, should always pass, all the time,  
and
if a test starts failing after you've changed code, you should assume  
you've
broken something and it deserves attention.

If necessary, we could/should have separate Makefile rules for testing  
these
two different groups of tests, so that we have an obvious 100% pass  
rate on
the tests in the main group of good tests.

-- Jon


On May 31, 2009, at 2:25 PM, Kelly O'Hair wrote:

> Cudos to Jon for having a testbase that is 100% pass, that IS the  
> way to go. ;^)
>
> So I think that means you can
>
>   cd langtools/test
>
> and either run jtreg, or use the Makefile
>
>   make PRODUCT_HOME=your_j2sdk-image-dir JT_HOME=jtreg_home TESTDIRS=.
>
> How you run the jtreg tests is up to you.
>
> See http://openjdk.java.net/jtreg/faq.html
>
> -kto
>
> Jonathan Gibbons wrote:
>> There are ~1500 tests in langtools/test, and all should pass.
>> -- Jon
>> On May 31, 2009, at 11:39 AM, Kelly O'Hair wrote:
>>> See  http://openjdk.java.net/jtreg/
>>>
>>>
>>> I tend to use the jdk/test/Makefile myself:
>>>
>>>  cd jdk/test && make PRODUCT_HOME=your_j2sdk-image-dir  
>>> JT_HOME=jtreg_home TESTDIRS=java/lang
>>>
>>> There are quite a few tests in the jdk/test directory and the above
>>> only runs the java/lang tests, which should all pass.
>>>
>>> As to which tests reliably pass and which fail, I can't tell you.
>>> Maybe someone else can refer to something that documents what tests
>>> are expected to pass in their area.
>>>
>>> -kto
>>>
>>> lianshun Liu wrote:
>>>> Hi,
>>>> When I've finished building OpenJDK, how can I make sure that it  
>>>> will work well? Is there any test suites that can be used to test  
>>>> the JVM?
>>>> Thanks :)
>>>> Lianshun Liu