[icedtea-web] RFC: add unit tests for the jnlp parser

[Dr Andrew John Hughes]

On 12:49 Fri 25 Mar     , Omair Majid wrote:

snip...

> >
> > I seem to recall that there was an option to alter JUnit output by implementing
> > some class but my information is pretty out of date.
> >
> 
> That's pretty much what I needed up doing. The only problem is 
> licensing. What license is that code under? JUnit is under Common Public 
> License and my custom code (attached) includes a few lines from JUnit 
> itself.
> 

Oh, something that still uses CPL.

There is a brief summary on Wikipedia: https://secure.wikimedia.org/wikipedia/en/wiki/Common_Public_License

Basically, it's the predecessor to EPL, the license Eclipse is under.

IANAL, but I don't see the problem with this.  The code is independent
from everything else in the codebase, isn't it?  So GPL
incompatibility isn't an issue (and we have the Classpath exception
anyway).  And it's a FOSS license so we can certainly include the
code.

What JUnit code is included?  Is it marked?

> > One option to consider is Mauve.  If we reused its test harness, any
> > improvements would also feed back to that project and benefit us with
> > JDK testing too.  The downside is that it isn't packaged, but I think
> > it's small enough that we could bring it into the IcedTea-Web tree.
> > Long-term, we could separate the harness itself out and use it with
> > both IcedTea-Web and Mauve.
> >
> 
> Using mauve would make it harder for new developers to contribute. Not 
> to mention that testing frameworks like JUnit and TestNG have much 
> better IDE integration.
> 

I didn't know IDE integration was a required feature, but you make a fair point.

I guess it was a bad suggestion.  I have the ulterior motive of
wanting to drive some work on Mauve.  It's a good test suite and one
which is used by a variety of projects, unlike the OpenJDK JTreg tests
which tend to be tightly attached to OpenJDK, both physically (files
in the same repo) and in the code, via reliance on HotSpot features
and internal classes.  Plus, it's a lot easier to get running than the
mess we currently have with JTreg.

> I would like to see less things in-tree (/me is looking at NanoXML), not 
> more.
> 

How is the NanoXML issue progressing?  It would be good to have a status update
on things and the progress towards the various releases of IcedTea-Web now 1.0
is done.

> >> I can work around it by writing a front end to it (already done, but I
> >> am not sure about the licensing here), or using something like ant to
> >> generate xml output which can be transformed (similar to how I have done
> >> it for testng) to text.
> >>
> >
> > Why is ant needed?   Please, no :-)
> >
> 
> If we want to avoid modifying junit and still want xml reports, it is 
> probably the only sane option.
> 

I really don't get want ant has to do with XML, given the XML APIs are in
the JDK.  If it's ant vs. these other dependencies for testng, I'd choose
the latter but that's a choice between a rock and a hard place, really.

Ant massively overcomplicates things, but I have no idea of the availability
of the various dependencies for testng or their licenses.

Can you provide some of this info?

> >> The second thing I tried was testng. This, of course, ends up requiring
> >> a number of jars that the builders may not have, though most
> >> distributions have testng packages.
> >>
> >> Any preferences on what to do? Should I just give up on formatting text
> >> results to be more jtreg-like? That would make it much simpler (and we
> >> will be able to use junit). Any other suggestions for test frameworks or
> >> how to make this work?
> >>
> >
> > If you could provide some example output from what you've tried, that would help.
> >
> 
> Using plain JUnit4:
> JUnit version 4.6
> .E
> Time: 0.004
> There was 1 failure:
> 1) initializationError(net.sourceforge.jnlp.ParserBasic)
> java.lang.NoClassDefFoundError: net/sourceforge/jnlp/ParseException
> 	at java.lang.Class.getDeclaredMethods0(Native Method)
> [snip]
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> 	... 20 more
> FAILURES!!!
> Tests run: 1,  Failures: 1
> 
> My custom runner:
> Passed: net.sourceforge.jnlp.ParserBasic.testResourcesJar
> Passed: net.sourceforge.jnlp.ParserBasic.testResourcesExtensions
> Passed: net.sourceforge.jnlp.ParserBasic.testResourcesProperty
> [snip]
> FAILED: testUnquotedAttributes(net.sourceforge.jnlp.ParserMalformedXml) 
> Invalid XML document syntax.
> Test results: passed: 23; failed: 4; ignored: 0
> 
> I suppose using plain JUnit is fine; it just wont have pretty output 
> suitable for buildbot.
> 

The custom runner looks fine.  What's the problem?

> Cheers,
> Omair

> diff -r d3edae4807f0 tests/junit-runner/CommandLine.java
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/tests/junit-runner/CommandLine.java	Tue Mar 08 18:12:16 2011 -0500
> @@ -0,0 +1,42 @@
> +import java.util.ArrayList;
> +import java.util.List;
> +
> +import org.junit.internal.JUnitSystem;
> +import org.junit.internal.RealSystem;
> +import org.junit.runner.JUnitCore;
> +import org.junit.runner.Result;
> +import org.junit.runner.notification.Failure;
> +import org.junit.runner.notification.RunListener;
> +
> +public class CommandLine extends JUnitCore {
> +
> +    public static void main(String... args) {
> +        runMainAndExit(new RealSystem(), args);
> +    }
> +
> +    public static void runMainAndExit(JUnitSystem system, String... args) {
> +        new CommandLine().runMain(system, args);
> +        system.exit(0);
> +    }
> +
> +    @Override
> +    public Result runMain(JUnitSystem system, String... args) {
> +        List<Class<?>> classes= new ArrayList<Class<?>>();
> +        List<Failure> missingClasses= new ArrayList<Failure>();
> +        for (String each : args) {
> +            try {
> +                classes.add(Class.forName(each));
> +            } catch (ClassNotFoundException e) {
> +                system.out().println("ERROR: Could not find class: " + each);
> +            }
> +        }
> +        RunListener listener= new LessVerboseTextListener(system);
> +        addListener(listener);
> +        Result result= run(classes.toArray(new Class[0]));
> +        for (Failure each : missingClasses) {
> +            result.getFailures().add(each);
> +        }
> +        return result;
> +    }
> +
> +}
> diff -r d3edae4807f0 tests/junit-runner/LessVerboseTextListener.java
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/tests/junit-runner/LessVerboseTextListener.java	Tue Mar 08 18:12:16 2011 -0500
> @@ -0,0 +1,44 @@
> +import java.io.PrintStream;
> +
> +import org.junit.internal.JUnitSystem;
> +import org.junit.runner.Description;
> +import org.junit.runner.Result;
> +import org.junit.runner.notification.Failure;
> +import org.junit.runner.notification.RunListener;
> +
> +public class LessVerboseTextListener extends RunListener {
> +
> +    private PrintStream writer;
> +    private boolean testFailed = false;
> +
> +    public LessVerboseTextListener(JUnitSystem system) {
> +        writer= system.out();
> +    }
> +
> +    @Override
> +    public void testStarted(Description description) throws Exception {
> +        testFailed = false;
> +    }
> +
> +    @Override
> +    public void testFailure(Failure failure) {
> +        testFailed = true;
> +        writer.println("FAILED: " + failure.getTestHeader() + " " + failure.getMessage());
> +    }
> +
> +    @Override
> +    public void testFinished(org.junit.runner.Description description) throws Exception {
> +        if (!testFailed) {
> +            writer.println("Passed: " + description.getClassName() + "." + description.getMethodName());
> +        }
> +    }
> +
> +    @Override
> +    public void testRunFinished(Result result) throws Exception {
> +        int passed = result.getRunCount() - result.getFailureCount() - result.getIgnoreCount();
> +        int failed = result.getFailureCount();
> +        int ignored = result.getIgnoreCount();
> +        writer.println("Test results: passed: " + passed + "; failed: " + failed + "; ignored: " + ignored);
> +    }
> +
> +}
> diff -r d3edae4807f0 tests/junit-runner/README
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/tests/junit-runner/README	Tue Mar 08 18:12:16 2011 -0500
> @@ -0,0 +1,3 @@
> +junit-runner is used to run tests instead of the standard runner
> +org.junit.runner.JUnitCore.  It provides output similar to that used by JTreg,
> +which is useful for automated comparison.


-- 
Andrew :)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

Support Free Java!
Contribute to GNU Classpath and IcedTea
http://www.gnu.org/software/classpath
http://icedtea.classpath.org
PGP Key: F5862A37 (https://keys.indymedia.org/)
Fingerprint = EA30 D855 D50F 90CD F54D  0698 0713 C3ED F586 2A37