[icedtea-web] RFC: add unit tests for the jnlp parser

[Dr Andrew John Hughes]

On 10:16 Thu 24 Mar     , Omair Majid wrote:
> On 03/23/2011 08:45 PM, Dr Andrew John Hughes wrote:
> > On 16:26 Wed 23 Mar     , Deepak Bhole wrote:
> >> * Omair Majid<omajid at redhat.com>  [2011-03-22 15:09]:
> >>> The unit tests rely on the testng framework. The patch adds optional
> >>> dependencies on testng, qdox and bsh for running these unit tests. I
> >>> have also added a dependency on xlst for transforming the test
> >>> output into a more jtreg-like format (useful for the buildbots)
> >>>
> >>> I also looked into using junit for these tests, but junit's command
> >>> line output is rather limited. To generate xml output, ant is
> >>> required as well.
> 
> > Well, before I even look at the patch, I want to be sure all this additional
> > crap is needed.  I've never even heard of any of these and having them all
> > as requirements is going to make it less likely that people will run tests.
> >
> > Are there no simpler solutions?  I do remember junit was fairly self-contained
> > the last time I used it, but that was circa 2003.
> 
> I am not too familiar with all the testing frameworks out there. I have 
> only used junit myself (and now I have dabbled in testng). These tests 
> have two target audiences - those building icedtea-web (especially the 
> buildbots) and programmers. Both need command line output to tell if the 
> tests are failing/passing.
> 

Agreed.  My main concern for the latter group (programmers) is the ease
of getting a working test environment set up i.e. what's the availability
of the test framework and its dependencies in various distributions?

> Naturally, I tried out junit first. I ran into an issue: it doesn't have 
> any good textual output (and does not support any formatting, though its 
> IDE-integration is quite good). It only shows tests failures, and a one 
> line summary for the entire test suite. There is no way to get 
> jtreg-like output. This is probably more of a concern for the buildbot 
> than anything else.
> 

I seem to recall that there was an option to alter JUnit output by implementing
some class but my information is pretty out of date.

One option to consider is Mauve.  If we reused its test harness, any
improvements would also feed back to that project and benefit us with
JDK testing too.  The downside is that it isn't packaged, but I think
it's small enough that we could bring it into the IcedTea-Web tree.
Long-term, we could separate the harness itself out and use it with
both IcedTea-Web and Mauve.

> I can work around it by writing a front end to it (already done, but I 
> am not sure about the licensing here), or using something like ant to 
> generate xml output which can be transformed (similar to how I have done 
> it for testng) to text.
> 

Why is ant needed?   Please, no :-)

> The second thing I tried was testng. This, of course, ends up requiring 
> a number of jars that the builders may not have, though most 
> distributions have testng packages.
> 
> Any preferences on what to do? Should I just give up on formatting text 
> results to be more jtreg-like? That would make it much simpler (and we 
> will be able to use junit). Any other suggestions for test frameworks or 
> how to make this work?
> 

If you could provide some example output from what you've tried, that would help.

> Thanks,
> Omair

-- 
Andrew :)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

Support Free Java!
Contribute to GNU Classpath and IcedTea
http://www.gnu.org/software/classpath
http://icedtea.classpath.org
PGP Key: F5862A37 (https://keys.indymedia.org/)
Fingerprint = EA30 D855 D50F 90CD F54D  0698 0713 C3ED F586 2A37