jtreg testing integrated

[Jonathan Gibbons]

Martin,

(removed compiler-dev)

Your comments about unsymmetric runs are interesting.  jtdiff performs 
an n-way
comparison, and I'd want to keep that functionality.

The two use cases I had in mind were:

-- Given a set of nightly builds on a set of platforms, compare the 
results across
    all the platforms, and report differences

-- Given the same set of nightly builds on a set of platforms, for each 
platform
    perform a pair-wise comparison against the corresponding results 
last night/week/month.

I'll see about adding an option to specify a reference set of results, 
for your
"developer time" use case.

---

Separately, check out the options for handling @ignore tests. Even on 
older versions
of jtreg you can use "-k:!ignore" to ignore @ignore tests.  (This works 
because @ignore
tests are given an implicit "ignore" keyword.)   With later versions of 
jtreg, you can use
-Ignore:{quiet,error,run} to control how @ignore tests should be 
handled. Using this
option, you should be to get closer to the goal of "all tests should 
pass", meaning
that there are less failures and so less need to compare the output 
results with jtdiff.

-- Jon



Martin Buchholz wrote:
> Jonathan,
>
> Thanks for jtdiff.
>
> Suggestions:
>
> - The various help options are confusing.
>   Just print all the help available if given -help or -usage.
> - The usage should give a paragraph explaining what it does.
>   Not too much work; why, you've practically written the required
>   words in the quoted text below.
> - My first version of diff-javatest was symmetrical.  It printed any
>   difference between the two runs, in both directions.  Later I
>   realized that (at least my own) usage invariably had the notion
>   of a "reference" JDK and a "test" JDK.  I was interested in tests
>   run in the "test" JDK but not in the reference JDK, but not vice
>   versa; typically the reference JDK results were historical ones
>   produced by someone wearing a QA hat, and were more complete
>   than the ones in the test JDK, where results were more likely to be
>   part of an edit-compile-test cycle.
>
> Try printing the usage message from my own diff-javatest script,
> which should still be accessible inside Sun, in /java/tl, for example.
>
> Martin
>
> On Mon, May 19, 2008 at 9:52 AM, Jonathan Gibbons
> <Jonathan.Gibbons at sun.com> wrote:
>   
>> Martin,
>>
>> jtreg is now open source, as of just before JavaOne. See
>> http://openjdk.java.net/jtreg
>>
>> There is a small new utility called "jtdiff" that comes with jtreg that may
>> do what
>> you want.  It will do n-way comparison of any JavaTest-type results (meaning
>> jtreg, JCK, etc)
>> where each set of results can be given as a JavaTest work directory, report
>> directory,
>> or just the summary.txt file within a report directory. The output can be
>> plain text or HTML.
>>
>> jtdiff is within jtreg.jar, so the easiest way to invoke it is
>>        java -cp jtreg.jar com.sun.javatest.diff.Main <args>
>>
>> -- Jon
>>
>> On May 19, 2008, at 8:30 AM, Martin Buchholz wrote:
>>
>>     
>>> [+compiler-dev, jtreg-use]
>>>
>>> On Mon, May 19, 2008 at 7:56 AM, Andrew John Hughes
>>> <gnu_andrew at member.fsf.org> wrote:
>>>       
>>>> 2008/5/19 Mark Wielaard <mark at klomp.org>:
>>>>         
>>>>> make jtregcheck -k runs the testsuites of hotspot (4 tests, all PASS),
>>>>> langtools (1,342 PASS, 1 FAIL - the version check) and jdk (2,875 tests
>>>>> of which about 130 fail - rerunning tests now). corba, jaxp and jaxws
>>>>> don't come with any tests. This takes about 3 hours on my machine.
>>>>>           
>>> Once upon a time, I wrote a test that made sure the hotspot
>>> and jdk library's idea of the current version and supported targets
>>> were in sync.  Unfortunately, it is not a requirement on hotspot
>>> integrations that they pass this test, so the test starts failing whenever
>>> hotspot starts supporting the class file version number for the next
>>> major release.  At least this is a strong hint to the javac team to
>>> catch up soon by incrementing their supported targets, etc...
>>>
>>> I like a policy of "Read my lips; no new test failures" but OpenJDK
>>> is not quite there; we get test failure creep when changes in
>>> one component break another component's tests.
>>>
>>>       
>>>>> Most of the failures are because the host javaweb.sfbay.sun.com cannot
>>>>> be resolved.
>>>>>           
>>> The jtreg tests were originally designed to be run only by Sun JDK
>>> development and test engineers.  If someone can come up with a
>>> portable way of testing network services (like ftp clients) withoutjsqrdg
>>> setting up a dedicated machine with a well-known name, that
>>> would be good.  Alternatively, making the name of this
>>> machine configurable when jtreg is run would also
>>> be an improvement, and a much simpler one.  But the obvious
>>> idea of using environment variables doesn't work.  Most environment
>>> variables are not passed to the running java test program.
>>>
>>> If it's considered acceptable for IcedTea hackers to get their
>>> hands dirty with not-100%-free technology, y'all could try
>>> running the jtreg tests against IcedTea, vanilla OpenJDK7,
>>> OpenJDK6, and JDK 6u6, and comparing the test failures.
>>>
>>> I once wrote a script to compare two jtreg test runs, diff-javatest.
>>> Jonathan et al, could you work (with me) on releasing that as open source?
>>>
>>> Martin
>>>
>>>       
>>>>> But there are also some genuine failures in java.awt.color,
>>>>> jmx.snmp, javax.script, javax.print, ... so enough to do for
>>>>> enterprising hackers!
>>>>>
>>>>>           
>>