CodeTools proposal: "SigTest"

[Kevin Looney]

Name: SigTest

Summary: A collection of tools used to evaluate platform compatibility 
and API change.

Proposed by: Kevin Looney

Rationale:

SigTest is already an Open Source project, located under the ME 
OpenSource infrastructure:

https://sigtest.java.net/

This is not a proposal for a new project, it is a proposal for a 
transfer of the SigTest (java.net) project into the OpenJDK CodeTools 
community, and moderation under OpenJDK bylaws.

The SigTest open source project is a collection of tools, originally 
based on Oracle's commercial SigTest tools product. The SigTest tools 
can be used to compare APIs and to measure the test coverage of an API. 
The tools were originally created to assist in the creation of Java 
technology compatibility test suites (TCKs), but are also useful in the 
creation of other types of test suites and in the software development 
process. The SigTest project consists of the following tools.

The *Signature Test tool* - makes it easy to compare the signatures of 
two different implementations or different versions of the same API. 
When it compares different implementations of the same API, the tool 
verifies that all of the members are present, reports when new members 
are added, and checks the specified behavior of each API member. When it 
compares different versions of the same API, the tool checks that the 
old version can be replaced by the new one without adversely affecting 
existing clients of the API.

The *API Coverage tool* can be used to estimate the test coverage a test 
suite provides for an implementation of a specified API. It does this by 
determining how many public class members the test suite references 
within the API specification. The tool uses a signature file 
representation of the API specification as the source of specification 
analysis. It does not process a formal specification in any form.

The *API Check tool* tracks API changes and roughly checks for source 
and binary compatibility. Much like Signature Test tool, it compares the 
reference implementation of an API recorded as golden signature file 
with a tested implementation. Unlike Signature Test tool, which requires 
that all classes (and the classes they depend on) be specified to the 
tool, API Check tool does not require dependencies to be specified. For 
that reason, API Check tool is faster than Signature Test tool, but less 
rigorous.

The SigTest open source project was created in 2008 - in order to 
develop a community that will improve the tools, improve software 
development techniques with respect to platform compatibility, and 
enhance the development of software test suites. We expect to continue 
this charter through inclusion in OpenJDK.

-- 
kevin.looney at oracle.com