[icedtea-web] RFC: add unit tests for the jnlp parser

[Omair Majid]

On 03/25/2011 11:07 AM, Dr Andrew John Hughes wrote:
> On 10:16 Thu 24 Mar     , Omair Majid wrote:
>> On 03/23/2011 08:45 PM, Dr Andrew John Hughes wrote:
>>> On 16:26 Wed 23 Mar     , Deepak Bhole wrote:
>>>> * Omair Majid<omajid at redhat.com>   [2011-03-22 15:09]:
>>>>> The unit tests rely on the testng framework. The patch adds optional
>>>>> dependencies on testng, qdox and bsh for running these unit tests. I
>>>>> have also added a dependency on xlst for transforming the test
>>>>> output into a more jtreg-like format (useful for the buildbots)
>>>>>
>>>>> I also looked into using junit for these tests, but junit's command
>>>>> line output is rather limited. To generate xml output, ant is
>>>>> required as well.
>>
>>> Well, before I even look at the patch, I want to be sure all this additional
>>> crap is needed.  I've never even heard of any of these and having them all
>>> as requirements is going to make it less likely that people will run tests.
>>>
>>> Are there no simpler solutions?  I do remember junit was fairly self-contained
>>> the last time I used it, but that was circa 2003.
>>
>> I am not too familiar with all the testing frameworks out there. I have
>> only used junit myself (and now I have dabbled in testng). These tests
>> have two target audiences - those building icedtea-web (especially the
>> buildbots) and programmers. Both need command line output to tell if the
>> tests are failing/passing.
>>
>
> Agreed.  My main concern for the latter group (programmers) is the ease
> of getting a working test environment set up i.e. what's the availability
> of the test framework and its dependencies in various distributions?
>
>> Naturally, I tried out junit first. I ran into an issue: it doesn't have
>> any good textual output (and does not support any formatting, though its
>> IDE-integration is quite good). It only shows tests failures, and a one
>> line summary for the entire test suite. There is no way to get
>> jtreg-like output. This is probably more of a concern for the buildbot
>> than anything else.
>>
>
> I seem to recall that there was an option to alter JUnit output by implementing
> some class but my information is pretty out of date.
>

That's pretty much what I needed up doing. The only problem is 
licensing. What license is that code under? JUnit is under Common Public 
License and my custom code (attached) includes a few lines from JUnit 
itself.

> One option to consider is Mauve.  If we reused its test harness, any
> improvements would also feed back to that project and benefit us with
> JDK testing too.  The downside is that it isn't packaged, but I think
> it's small enough that we could bring it into the IcedTea-Web tree.
> Long-term, we could separate the harness itself out and use it with
> both IcedTea-Web and Mauve.
>

Using mauve would make it harder for new developers to contribute. Not 
to mention that testing frameworks like JUnit and TestNG have much 
better IDE integration.

I would like to see less things in-tree (/me is looking at NanoXML), not 
more.

>> I can work around it by writing a front end to it (already done, but I
>> am not sure about the licensing here), or using something like ant to
>> generate xml output which can be transformed (similar to how I have done
>> it for testng) to text.
>>
>
> Why is ant needed?   Please, no :-)
>

If we want to avoid modifying junit and still want xml reports, it is 
probably the only sane option.

>> The second thing I tried was testng. This, of course, ends up requiring
>> a number of jars that the builders may not have, though most
>> distributions have testng packages.
>>
>> Any preferences on what to do? Should I just give up on formatting text
>> results to be more jtreg-like? That would make it much simpler (and we
>> will be able to use junit). Any other suggestions for test frameworks or
>> how to make this work?
>>
>
> If you could provide some example output from what you've tried, that would help.
>

Using plain JUnit4:
JUnit version 4.6
.E
Time: 0.004
There was 1 failure:
1) initializationError(net.sourceforge.jnlp.ParserBasic)
java.lang.NoClassDefFoundError: net/sourceforge/jnlp/ParseException
	at java.lang.Class.getDeclaredMethods0(Native Method)
[snip]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
	... 20 more
FAILURES!!!
Tests run: 1,  Failures: 1

My custom runner:
Passed: net.sourceforge.jnlp.ParserBasic.testResourcesJar
Passed: net.sourceforge.jnlp.ParserBasic.testResourcesExtensions
Passed: net.sourceforge.jnlp.ParserBasic.testResourcesProperty
[snip]
FAILED: testUnquotedAttributes(net.sourceforge.jnlp.ParserMalformedXml) 
Invalid XML document syntax.
Test results: passed: 23; failed: 4; ignored: 0

I suppose using plain JUnit is fine; it just wont have pretty output 
suitable for buildbot.

Cheers,
Omair
-------------- next part --------------
A non-text attachment was scrubbed...
Name: custom-junit-runner.patch
Type: text/x-patch
Size: 3603 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/distro-pkg-dev/attachments/20110325/f410fe4c/custom-junit-runner.patch