jtreg testing integrated

Andrew John Hughes gnu_andrew at member.fsf.org
Wed May 21 14:14:23 PDT 2008


On 21/05/2008, Joseph D. Darcy <Joe.Darcy at sun.com> wrote:
> Hello Mark.
>
>  Mark Wielaard wrote:
>
> > Hi Joe,
> >
> > On Mon, 2008-05-19 at 09:59 -0700, Joseph D. Darcy wrote:
> >
> >
> > > Mark Wielaard wrote:
> > >
> > >
> > > > On Mon, 2008-05-19 at 11:21 +0200, Mark Wielaard wrote:
> > > >
> > > >
> > > > > make jtregcheck -k runs the testsuites of hotspot (4 tests, all
> PASS),
> > > > > langtools (1,342 PASS, 1 FAIL - the version check) and jdk (2,875
> tests
> > > > > of which about 130 fail - rerunning tests now). corba, jaxp and
> jaxws
> > > > > don't come with any tests. This takes about 3 hours on my machine.
> > > > >
> > > > >
> > > > Uploaded the text summary files if someone wants to compare results,
> or
> > > > if someone is looking for a test failure to work on:
> > > > http://icedtea.classpath.org/~mjw/jtreg/
> > > >
> > > > The final result for jdk was:
> > > > Test results: passed: 2,658; failed: 134; error: 7
> > > >
> > > >
> > > Last week, I started taking another look at the jtreg test results
> inside Sun.  As has been noted, some of the tests are invalid for OpenJDK 6.
>  I've started correcting them and some fixes will be in the next source
> drop.
> > >
> > >
> >
> > That is great. Do you have results published somewhere already to
> > compare with?
> >
>
>  I don't have tests published at the moment, but I'll publish results on my
> blog (http://blogs.sun.com/darcy) for the next build, b10, which should be
> arriving soon.  For b09, on a Solaris SPARC run I got around regression 65
> test failures on OpenJDK 6; some of tests have since been fixed and other
> failures should be resolved by work that was been done in b10.  (The tests
> relying on the Sun-internal servers pass, but that isn't too helpful to
> people outside Sun's firewall :-)
>
>
> >  Are there any tests you need help with or don't have time
> > for at the moment?
> >
> >
>
>  Thanks for the offer!  Let's look at this again after the next build.
>

That's the benefit of community involvement :)

>
> > I think the most important thing is making sure the tests don't depend
> > on any sun internal only servers. That seems to solve the largest part
> > of the failures we are currently seeing.
> >
> >
>
>  Those may take a little while to figure out a good fix in the sense of
> still testing the networking code outside Sun's firewall as opposed to just
> adding "if(!insideSun()) pass" logic.  I'll talk to the networking team
> about this.
>

Well I don't believe anyone was suggesting such a simplistic hack.  We
need a well-known external host with good uptime that would handle
such requests without breaking a sweat.

>  On a related note, there are times where it could be useful for a test to
> have somewhat different behavior on Sun's production JDK versus OpenJDK.
> For example, I'm the author of the
> java/lang/reflect/Generics/Probe.java test which fails on
> OpenJDK since it references some production-only classes.  There are a few
> options to fix this.  One way would be to remove all reference to the
> production-only classes, which would reduce the usefulness of the test for
> the production code; conversely, the whole test could be moved to closed,
> reducing the test coverage of OpenJDK.  The test could be split into open
> and closed parts, complicating maintenance.  So instead I changed the test
> to only look at the production classes for a production build.  I test for a
> production build using
>
>  System.getProperty("java.runtime.name").startsWith("Java(TM)")
>
>  Out of the box, our OpenJDK builds have "java.runtime.name" start with
> "OpenJDK"; is that true in your IcedTea/OpenJDK builds too?
>

I assume what you mean by a 'production build' is the code being used
to build a proprietary JDK with additional non-Free classes.  If this
is the case, I would hope that, in the long term, these will no longer
be needed and the 'production build' will be just a build from the
GPLed codebase like any other.  I mention this because I think the
term could be a little disparaging to outsiders, as it implies only a
build with proprietary Sun code is considered production ready.

I wouldn't add any code to rely on the name of a build from the GPLed
code.  It's just asking for trouble further down the line.  Besides,
by including such tests, you're effectively wrongly favouring some
GPLed distributions over others.

Is a test for the production system (with an appropriate comment so we
know that's what it is) not sufficient?

>  For the javax/script tests, I was going to add "if (no javascript engine
> and production JDK) fail else note/warning" logic so that our production
> build gets the same coverage and if an OpenJDK build come with a javascript
> engine, it would get testing too.
>

Wouldn't testing the result of
ScriptEngineManager.getEngineFactories() suffice? Why do you need to
specifically test for a production build?

>  Cheers,
>
>  -Joe
>


-- 
Andrew :-)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA  7927 142C 2591 94EF D9D8



More information about the distro-pkg-dev mailing list