New Group Proposal: OpenJDK Conformance

[Andrew John Hughes]

On Monday 01 October 2007 08:58:57 Mark Wielaard wrote:
> Hi,
>
> On Fri, 2007-09-28 at 10:57 -0700, iris.clark at sun.com wrote:
> > I'd like to propose creation of the OpenJDK Conformance Group.  The
> > intent of this group is to discuss conformance testing and
> > compatibility issues.  Topics for discussion include:
> >
> >   - Defining conformance testing
> >   - Understanding the importance of compatibility between releases
> >   - Distinguishing between conformance and product tests
>
> This sound great. One idea for this is inviting Stuart Ballard (CCed).
> His japitools http://www.kaffe.org/~stuart/japi/ have been a great part
> of the communities conformance and compatibility drive.
>

Seconded; Stuart's tools have been invaluable during the development of GNU 
Classpath and are equally applicable as general tools for ensuring binary API 
compatabillity between releases.

> Does it have to be a separate group? Couldn't this be part of the
> quality group? They already have a (pretty quiet) mailinglist and
> infrastructure on the website. It seems conformance is just one bit of
> the quality process overall.
>

I agree, there are already an overly burdernsome number of mailing lists and 
groups and conformance would intrinsicly seem to be part of a drive for 
quality.

The quality page already has plenty of nice metrics:

http://openjdk.java.net/groups/quality/

It would seem an appropriate place for further conformance-related ones.

> >   - Acquiring the JCK for an OpenJDK project
> >   - Configuring and running the JCK
> >   - Contributing new JCK tests
> >
> > This group will have two mailing lists.  The first is open to everyone
> > and will contain discussions around compatibility, conformance
> > testing, and general JCK installation and usability.  The second
> > mailing list is open only to group members who have signed the OpenJDK
> > Community TCK Licensee Agreement.  Discussions on this list will focus
> > on understanding the behavior and validity of specific tests and other
> > confidential materials.
>
> I do have my doubts about this second part though. Is it really in the
> interest of the openjdk project to have secret lists where proprietary
> software is discussed without the rest of the community being able to
> see, share and help out? The thing I like about OpenJDK is that it is a
> free software community, where all software and ideas are shared in the
> open. There are still of course the binary blobs and the jtreg suite,
> but my understanding is that those will be liberated over time (as
> icedtea and mauve have shown can already be done). Is the idea that over
> time the JCK will also become a proper part of OpenJDK under a free
> license?
>

This is indicative of what my feelings about the OpenJDK project have been so 
far unfortunately.  I'd like, as would many GNU Classpath members, to see it 
blossom with a rich interacting community but this won't happen while large 
parts of the process remain behind closed doors.  I can understand if there 
needs to be some private discussion between licensees but the development of 
the test suite should be done in public, by the community, with the long term 
goal of an open test suite under a Free software license.  To my mind, the 
whole certification process should be just one element of the test suite, not 
its primary motivation.  We've developed Mauve because the JDK test suite was 
inaccessible in the past, and I hope this is not the way things are going to 
continue.  It would be nice to instead merge these efforts and work together 
to create the best possible test suite.

> Cheers,
>
> Mark

Cheers,
-- 
Andrew :)