jtreg testing integrated
Andrew John Hughes
gnu_andrew at member.fsf.org
Wed May 21 14:14:23 PDT 2008
On 21/05/2008, Joseph D. Darcy <Joe.Darcy at sun.com> wrote:
> Hello Mark.
>
> Mark Wielaard wrote:
>
> > Hi Joe,
> >
> > On Mon, 2008-05-19 at 09:59 -0700, Joseph D. Darcy wrote:
> >
> >
> > > Mark Wielaard wrote:
> > >
> > >
> > > > On Mon, 2008-05-19 at 11:21 +0200, Mark Wielaard wrote:
> > > >
> > > >
> > > > > make jtregcheck -k runs the testsuites of hotspot (4 tests, all
> PASS),
> > > > > langtools (1,342 PASS, 1 FAIL - the version check) and jdk (2,875
> tests
> > > > > of which about 130 fail - rerunning tests now). corba, jaxp and
> jaxws
> > > > > don't come with any tests. This takes about 3 hours on my machine.
> > > > >
> > > > >
> > > > Uploaded the text summary files if someone wants to compare results,
> or
> > > > if someone is looking for a test failure to work on:
> > > > http://icedtea.classpath.org/~mjw/jtreg/
> > > >
> > > > The final result for jdk was:
> > > > Test results: passed: 2,658; failed: 134; error: 7
> > > >
> > > >
> > > Last week, I started taking another look at the jtreg test results
> inside Sun. As has been noted, some of the tests are invalid for OpenJDK 6.
> I've started correcting them and some fixes will be in the next source
> drop.
> > >
> > >
> >
> > That is great. Do you have results published somewhere already to
> > compare with?
> >
>
> I don't have tests published at the moment, but I'll publish results on my
> blog (http://blogs.sun.com/darcy) for the next build, b10, which should be
> arriving soon. For b09, on a Solaris SPARC run I got around regression 65
> test failures on OpenJDK 6; some of tests have since been fixed and other
> failures should be resolved by work that was been done in b10. (The tests
> relying on the Sun-internal servers pass, but that isn't too helpful to
> people outside Sun's firewall :-)
>
>
> > Are there any tests you need help with or don't have time
> > for at the moment?
> >
> >
>
> Thanks for the offer! Let's look at this again after the next build.
>
That's the benefit of community involvement :)
>
> > I think the most important thing is making sure the tests don't depend
> > on any sun internal only servers. That seems to solve the largest part
> > of the failures we are currently seeing.
> >
> >
>
> Those may take a little while to figure out a good fix in the sense of
> still testing the networking code outside Sun's firewall as opposed to just
> adding "if(!insideSun()) pass" logic. I'll talk to the networking team
> about this.
>
Well I don't believe anyone was suggesting such a simplistic hack. We
need a well-known external host with good uptime that would handle
such requests without breaking a sweat.
> On a related note, there are times where it could be useful for a test to
> have somewhat different behavior on Sun's production JDK versus OpenJDK.
> For example, I'm the author of the
> java/lang/reflect/Generics/Probe.java test which fails on
> OpenJDK since it references some production-only classes. There are a few
> options to fix this. One way would be to remove all reference to the
> production-only classes, which would reduce the usefulness of the test for
> the production code; conversely, the whole test could be moved to closed,
> reducing the test coverage of OpenJDK. The test could be split into open
> and closed parts, complicating maintenance. So instead I changed the test
> to only look at the production classes for a production build. I test for a
> production build using
>
> System.getProperty("java.runtime.name").startsWith("Java(TM)")
>
> Out of the box, our OpenJDK builds have "java.runtime.name" start with
> "OpenJDK"; is that true in your IcedTea/OpenJDK builds too?
>
I assume what you mean by a 'production build' is the code being used
to build a proprietary JDK with additional non-Free classes. If this
is the case, I would hope that, in the long term, these will no longer
be needed and the 'production build' will be just a build from the
GPLed codebase like any other. I mention this because I think the
term could be a little disparaging to outsiders, as it implies only a
build with proprietary Sun code is considered production ready.
I wouldn't add any code to rely on the name of a build from the GPLed
code. It's just asking for trouble further down the line. Besides,
by including such tests, you're effectively wrongly favouring some
GPLed distributions over others.
Is a test for the production system (with an appropriate comment so we
know that's what it is) not sufficient?
> For the javax/script tests, I was going to add "if (no javascript engine
> and production JDK) fail else note/warning" logic so that our production
> build gets the same coverage and if an OpenJDK build come with a javascript
> engine, it would get testing too.
>
Wouldn't testing the result of
ScriptEngineManager.getEngineFactories() suffice? Why do you need to
specifically test for a production build?
> Cheers,
>
> -Joe
>
--
Andrew :-)
Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net
PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA 7927 142C 2591 94EF D9D8
More information about the distro-pkg-dev
mailing list