jtreg testing integrated

[Jonathan Gibbons]

On May 20, 2008, at 2:32 AM, Mark Wielaard wrote:

> Hi Martin,
>
> On Mon, 2008-05-19 at 08:30 -0700, Martin Buchholz wrote:
>> On Mon, May 19, 2008 at 7:56 AM, Andrew John Hughes
>> <gnu_andrew at member.fsf.org> wrote:
>>> 2008/5/19 Mark Wielaard <mark at klomp.org>:
>>>>
>>>> make jtregcheck -k runs the testsuites of hotspot (4 tests, all  
>>>> PASS),
>>>> langtools (1,342 PASS, 1 FAIL - the version check) and jdk (2,875  
>>>> tests
>>>> of which about 130 fail - rerunning tests now). corba, jaxp and  
>>>> jaxws
>>>> don't come with any tests. This takes about 3 hours on my machine.
>>
>> Once upon a time, I wrote a test that made sure the hotspot
>> and jdk library's idea of the current version and supported targets
>> were in sync.  Unfortunately, it is not a requirement on hotspot
>> integrations that they pass this test, so the test starts failing  
>> whenever
>> hotspot starts supporting the class file version number for the next
>> major release.  At least this is a strong hint to the javac team to
>> catch up soon by incrementing their supported targets, etc...
>
> In this case it is a more mundane version check failure:
> tools/javac/versions/check.sh
> javac thinks its version isn't 1.6.0, but javac 1.6.0-internal
>
>> I like a policy of "Read my lips; no new test failures" but OpenJDK
>> is not quite there; we get test failure creep when changes in
>> one component break another component's tests.
>
> Yes, that would be idea. At least for openjdk6/icedtea we seem to be
> pretty close actually. It will be more challenging for openjdk7. I
> haven't quite figured out all the dynamics around "workspace
> integration". But I assume we can get the master tree to zero fail and
> then demand that any integration cycle doesn't introduce regressions.
>
>> The jtreg tests were originally designed to be run only by Sun JDK
>> development and test engineers.  If someone can come up with a
>> portable way of testing network services (like ftp clients) without
>> setting up a dedicated machine with a well-known name, that
>> would be good.  Alternatively, making the name of this
>> machine configurable when jtreg is run would also
>> be an improvement, and a much simpler one.  But the obvious
>> idea of using environment variables doesn't work.  Most environment
>> variables are not passed to the running java test program.
>
> Making it configurable, or even ignorable with keywords would be  
> crucial
> for distribution testing. Most distributions don't allow their build
> daemons to access the network. But for quality control it is essential
> that they do run the full test suite.

jtreg allows tests to be tagged with keyords, that can be used on the  
command
line to filter the tests to be executed.

>
>
> I haven't made an inventory of what services would be needed on a
> machine to be replace javaweb.sfbay.sun.com with a public machine, but
> we can certainly run some services on icedtea.classpath.org or maybe  
> one
> of the openjdk.java.net machines.
>
>> If it's considered acceptable for IcedTea hackers to get their
>> hands dirty with not-100%-free technology, y'all could try
>> running the jtreg tests against IcedTea, vanilla OpenJDK7,
>> OpenJDK6, and JDK 6u6, and comparing the test failures.
>
> :) Well, the whole idea behind IcedTea is to provide an OpenJDK
> derivative that doesn't depend on any non-free build or runtime
> requirements.
>
> But I am certainly interested in comparing results. I do think  
> OpenJDK6
> and IcedTea are now so close that we shouldn't be seeing any test  
> result
> differences between the two.
>
> That brings up the question how to export a JTreport/JTwork  
> environment.
> I only posted the text results http://icedtea.classpath.org/~mjw/ 
> jtreg/
> since the JTreport and JTwork files have all kinds of hard coded
> absolute path references. It would be nice to be able to export it all
> so I can upload it to some public site for others to look at and  
> compare
> with.

At a minimum, you'd want to publish the summary.txt files from the  
report directory.
Note also that JT Harness comes with a couple of servlets you can  
install for pretty
viewing .jtr and .jtx files.

>
>
> Cheers,
>
> MArk
>