jtreg testing integrated

[Martin Buchholz]

Jonathan,

Thanks for jtdiff.

Suggestions:

- The various help options are confusing.
  Just print all the help available if given -help or -usage.
- The usage should give a paragraph explaining what it does.
  Not too much work; why, you've practically written the required
  words in the quoted text below.
- My first version of diff-javatest was symmetrical.  It printed any
  difference between the two runs, in both directions.  Later I
  realized that (at least my own) usage invariably had the notion
  of a "reference" JDK and a "test" JDK.  I was interested in tests
  run in the "test" JDK but not in the reference JDK, but not vice
  versa; typically the reference JDK results were historical ones
  produced by someone wearing a QA hat, and were more complete
  than the ones in the test JDK, where results were more likely to be
  part of an edit-compile-test cycle.

Try printing the usage message from my own diff-javatest script,
which should still be accessible inside Sun, in /java/tl, for example.

Martin

On Mon, May 19, 2008 at 9:52 AM, Jonathan Gibbons
<Jonathan.Gibbons at sun.com> wrote:
> Martin,
>
> jtreg is now open source, as of just before JavaOne. See
> http://openjdk.java.net/jtreg
>
> There is a small new utility called "jtdiff" that comes with jtreg that may
> do what
> you want.  It will do n-way comparison of any JavaTest-type results (meaning
> jtreg, JCK, etc)
> where each set of results can be given as a JavaTest work directory, report
> directory,
> or just the summary.txt file within a report directory. The output can be
> plain text or HTML.
>
> jtdiff is within jtreg.jar, so the easiest way to invoke it is
>        java -cp jtreg.jar com.sun.javatest.diff.Main <args>
>
> -- Jon
>
> On May 19, 2008, at 8:30 AM, Martin Buchholz wrote:
>
>> [+compiler-dev, jtreg-use]
>>
>> On Mon, May 19, 2008 at 7:56 AM, Andrew John Hughes
>> <gnu_andrew at member.fsf.org> wrote:
>>>
>>> 2008/5/19 Mark Wielaard <mark at klomp.org>:
>>>>
>>>> make jtregcheck -k runs the testsuites of hotspot (4 tests, all PASS),
>>>> langtools (1,342 PASS, 1 FAIL - the version check) and jdk (2,875 tests
>>>> of which about 130 fail - rerunning tests now). corba, jaxp and jaxws
>>>> don't come with any tests. This takes about 3 hours on my machine.
>>
>> Once upon a time, I wrote a test that made sure the hotspot
>> and jdk library's idea of the current version and supported targets
>> were in sync.  Unfortunately, it is not a requirement on hotspot
>> integrations that they pass this test, so the test starts failing whenever
>> hotspot starts supporting the class file version number for the next
>> major release.  At least this is a strong hint to the javac team to
>> catch up soon by incrementing their supported targets, etc...
>>
>> I like a policy of "Read my lips; no new test failures" but OpenJDK
>> is not quite there; we get test failure creep when changes in
>> one component break another component's tests.
>>
>>>> Most of the failures are because the host javaweb.sfbay.sun.com cannot
>>>> be resolved.
>>
>> The jtreg tests were originally designed to be run only by Sun JDK
>> development and test engineers.  If someone can come up with a
>> portable way of testing network services (like ftp clients) without
>> setting up a dedicated machine with a well-known name, that
>> would be good.  Alternatively, making the name of this
>> machine configurable when jtreg is run would also
>> be an improvement, and a much simpler one.  But the obvious
>> idea of using environment variables doesn't work.  Most environment
>> variables are not passed to the running java test program.
>>
>> If it's considered acceptable for IcedTea hackers to get their
>> hands dirty with not-100%-free technology, y'all could try
>> running the jtreg tests against IcedTea, vanilla OpenJDK7,
>> OpenJDK6, and JDK 6u6, and comparing the test failures.
>>
>> I once wrote a script to compare two jtreg test runs, diff-javatest.
>> Jonathan et al, could you work (with me) on releasing that as open source?
>>
>> Martin
>>
>>>> But there are also some genuine failures in java.awt.color,
>>>> jmx.snmp, javax.script, javax.print, ... so enough to do for
>>>> enterprising hackers!
>>>>
>
>