jtreg testing integrated

[Joseph D. Darcy]

Hello Mark.

Mark Wielaard wrote:
> Hi Joe,
>
> On Mon, 2008-05-19 at 09:59 -0700, Joseph D. Darcy wrote:
>   
>> Mark Wielaard wrote:
>>     
>>> On Mon, 2008-05-19 at 11:21 +0200, Mark Wielaard wrote:
>>>       
>>>> make jtregcheck -k runs the testsuites of hotspot (4 tests, all PASS),
>>>> langtools (1,342 PASS, 1 FAIL - the version check) and jdk (2,875 tests
>>>> of which about 130 fail - rerunning tests now). corba, jaxp and jaxws
>>>> don't come with any tests. This takes about 3 hours on my machine.
>>>>         
>>> Uploaded the text summary files if someone wants to compare results, or
>>> if someone is looking for a test failure to work on:
>>> http://icedtea.classpath.org/~mjw/jtreg/
>>>
>>> The final result for jdk was:
>>> Test results: passed: 2,658; failed: 134; error: 7
>>>       
>> Last week, I started taking another look at the jtreg test results 
>> inside Sun.  As has been noted, some of the tests are invalid for 
>> OpenJDK 6.  I've started correcting them and some fixes will be in the 
>> next source drop.
>>     
>
> That is great. Do you have results published somewhere already to
> compare with?

I don't have tests published at the moment, but I'll publish results on 
my blog (http://blogs.sun.com/darcy) for the next build, b10, which 
should be arriving soon.  For b09, on a Solaris SPARC run I got around 
regression 65 test failures on OpenJDK 6; some of tests have since been 
fixed and other failures should be resolved by work that was been done 
in b10.  (The tests relying on the Sun-internal servers pass, but that 
isn't too helpful to people outside Sun's firewall :-)

>  Are there any tests you need help with or don't have time
> for at the moment?
>   

Thanks for the offer!  Let's look at this again after the next build.

> I think the most important thing is making sure the tests don't depend
> on any sun internal only servers. That seems to solve the largest part
> of the failures we are currently seeing.
>   

Those may take a little while to figure out a good fix in the sense of 
still testing the networking code outside Sun's firewall as opposed to 
just adding "if(!insideSun()) pass" logic.  I'll talk to the networking 
team about this.

On a related note, there are times where it could be useful for a test 
to have somewhat different behavior on Sun's production JDK versus 
OpenJDK.  For example, I'm the author of the 
java/lang/reflect/Generics/Probe.java test which fails on OpenJDK since 
it references some production-only classes.  There are a few options to 
fix this.  One way would be to remove all reference to the 
production-only classes, which would reduce the usefulness of the test 
for the production code; conversely, the whole test could be moved to 
closed, reducing the test coverage of OpenJDK.  The test could be split 
into open and closed parts, complicating maintenance.  So instead I 
changed the test to only look at the production classes for a production 
build.  I test for a production build using

System.getProperty("java.runtime.name").startsWith("Java(TM)")

Out of the box, our OpenJDK builds have "java.runtime.name" start with 
"OpenJDK"; is that true in your IcedTea/OpenJDK builds too?

For the javax/script tests, I was going to add "if (no javascript engine 
and production JDK) fail else note/warning" logic so that our production 
build gets the same coverage and if an OpenJDK build come with a 
javascript engine, it would get testing too.

Cheers,

-Joe