jtreg testing integrated

[Jonathan Gibbons]

Martin,

jtreg is now open source, as of just before JavaOne. See http://openjdk.java.net/jtreg

There is a small new utility called "jtdiff" that comes with jtreg  
that may do what
you want.  It will do n-way comparison of any JavaTest-type results  
(meaning jtreg, JCK, etc)
where each set of results can be given as a JavaTest work directory,  
report directory,
or just the summary.txt file within a report directory. The output can  
be plain text or HTML.

jtdiff is within jtreg.jar, so the easiest way to invoke it is
	java -cp jtreg.jar com.sun.javatest.diff.Main <args>

-- Jon

On May 19, 2008, at 8:30 AM, Martin Buchholz wrote:

> [+compiler-dev, jtreg-use]
>
> On Mon, May 19, 2008 at 7:56 AM, Andrew John Hughes
> <gnu_andrew at member.fsf.org> wrote:
>> 2008/5/19 Mark Wielaard <mark at klomp.org>:
>>>
>>> make jtregcheck -k runs the testsuites of hotspot (4 tests, all  
>>> PASS),
>>> langtools (1,342 PASS, 1 FAIL - the version check) and jdk (2,875  
>>> tests
>>> of which about 130 fail - rerunning tests now). corba, jaxp and  
>>> jaxws
>>> don't come with any tests. This takes about 3 hours on my machine.
>
> Once upon a time, I wrote a test that made sure the hotspot
> and jdk library's idea of the current version and supported targets
> were in sync.  Unfortunately, it is not a requirement on hotspot
> integrations that they pass this test, so the test starts failing  
> whenever
> hotspot starts supporting the class file version number for the next
> major release.  At least this is a strong hint to the javac team to
> catch up soon by incrementing their supported targets, etc...
>
> I like a policy of "Read my lips; no new test failures" but OpenJDK
> is not quite there; we get test failure creep when changes in
> one component break another component's tests.
>
>>> Most of the failures are because the host javaweb.sfbay.sun.com  
>>> cannot
>>> be resolved.
>
> The jtreg tests were originally designed to be run only by Sun JDK
> development and test engineers.  If someone can come up with a
> portable way of testing network services (like ftp clients) without
> setting up a dedicated machine with a well-known name, that
> would be good.  Alternatively, making the name of this
> machine configurable when jtreg is run would also
> be an improvement, and a much simpler one.  But the obvious
> idea of using environment variables doesn't work.  Most environment
> variables are not passed to the running java test program.
>
> If it's considered acceptable for IcedTea hackers to get their
> hands dirty with not-100%-free technology, y'all could try
> running the jtreg tests against IcedTea, vanilla OpenJDK7,
> OpenJDK6, and JDK 6u6, and comparing the test failures.
>
> I once wrote a script to compare two jtreg test runs, diff-javatest.
> Jonathan et al, could you work (with me) on releasing that as open  
> source?
>
> Martin
>
>>> But there are also some genuine failures in java.awt.color,
>>> jmx.snmp, javax.script, javax.print, ... so enough to do for
>>> enterprising hackers!
>>>