jtreg testing integrated

[Mark Wielaard]

Hi Joe,

On Wed, 2008-05-21 at 11:37 -0700, Joseph D. Darcy wrote:
> > I think the most important thing is making sure the tests don't depend
> > on any sun internal only servers. That seems to solve the largest part
> > of the failures we are currently seeing.
> >   
> Those may take a little while to figure out a good fix in the sense of 
> still testing the networking code outside Sun's firewall as opposed to 
> just adding "if(!insideSun()) pass" logic.  I'll talk to the networking 
> team about this.

I did look through some of the failures and it looks we basically need
just 3 hosts:

  HOST = "javaweb.sfbay.sun.com";
  REFUSING_HOST = "jano.sfbay.sun.com";
  FAR_HOST = "theclub.ireland.sun.com";
  UNRESOLVABLE_HOST = "blah-blah.blah-blah.blah";

With daytime and echo available. The only wrinkle is that some tests
expect to be able to use quick timeouts from HOST, but to need long
timeouts from FAR_HOST. I think we could provide something public for
this. But I might have missed some tests that need other services. Do
you have a full overview?

> On a related note, there are times where it could be useful for a test 
> to have somewhat different behavior on Sun's production JDK versus 
> OpenJDK.  For example, I'm the author of the 
> java/lang/reflect/Generics/Probe.java test which fails on OpenJDK since 
> it references some production-only classes.  There are a few options to 
> fix this.  One way would be to remove all reference to the 
> production-only classes, which would reduce the usefulness of the test 
> for the production code; conversely, the whole test could be moved to 
> closed, reducing the test coverage of OpenJDK.  The test could be split 
> into open and closed parts, complicating maintenance.  So instead I 
> changed the test to only look at the production classes for a production 
> build.  I test for a production build using
> 
> System.getProperty("java.runtime.name").startsWith("Java(TM)")
> 
> Out of the box, our OpenJDK builds have "java.runtime.name" start with 
> "OpenJDK"; is that true in your IcedTea/OpenJDK builds too?

It depends on the version and the distribution. IcedTea6 now calls
itself OpenJDK, but IcedTea7 calls itself IcedTea. Ubuntu has a patch to
return "Java(TM)" here. And we might actually might want to make that
the default. There are already bug reports about applications and
applets testing for this string to be exactly in the way as the
proprietary implementations (you could argue that is a bug in that
application/applet, but users don't really care unfortunately).

Whenever possible I think we should make this configurable on a specific
test property. That is more work, but allows all derivatives, not just
Sun's ClosedJDK to use the same tests with their own settings. The
defaults should of course be what works with generic upstream OpenJDK.

I was looking at this particular test and I didn't immediately see why
javax.crypto.SunJCE_x couldn't be replaced with a commonly available
class in OpenJDK. What is special about those classes that they need to
be explicitly tested against?

> For the javax/script tests, I was going to add "if (no javascript engine 
> and production JDK) fail else note/warning" logic so that our production 
> build gets the same coverage and if an OpenJDK build come with a 
> javascript engine, it would get testing too.

What do you precisely mean when you say that you have an "if production
JDK"? What functional feature does that test against?

Cheers,

Mark