jtreg testing integrated

[Mark Wielaard]

Hi Martin,

On Mon, 2008-05-19 at 08:30 -0700, Martin Buchholz wrote:
> On Mon, May 19, 2008 at 7:56 AM, Andrew John Hughes
> <gnu_andrew at member.fsf.org> wrote:
> > 2008/5/19 Mark Wielaard <mark at klomp.org>:
> >>
> >> make jtregcheck -k runs the testsuites of hotspot (4 tests, all PASS),
> >> langtools (1,342 PASS, 1 FAIL - the version check) and jdk (2,875 tests
> >> of which about 130 fail - rerunning tests now). corba, jaxp and jaxws
> >> don't come with any tests. This takes about 3 hours on my machine.
> 
> Once upon a time, I wrote a test that made sure the hotspot
> and jdk library's idea of the current version and supported targets
> were in sync.  Unfortunately, it is not a requirement on hotspot
> integrations that they pass this test, so the test starts failing whenever
> hotspot starts supporting the class file version number for the next
> major release.  At least this is a strong hint to the javac team to
> catch up soon by incrementing their supported targets, etc...

In this case it is a more mundane version check failure:
tools/javac/versions/check.sh
javac thinks its version isn't 1.6.0, but javac 1.6.0-internal

> I like a policy of "Read my lips; no new test failures" but OpenJDK
> is not quite there; we get test failure creep when changes in
> one component break another component's tests.

Yes, that would be idea. At least for openjdk6/icedtea we seem to be
pretty close actually. It will be more challenging for openjdk7. I
haven't quite figured out all the dynamics around "workspace
integration". But I assume we can get the master tree to zero fail and
then demand that any integration cycle doesn't introduce regressions.

> The jtreg tests were originally designed to be run only by Sun JDK
> development and test engineers.  If someone can come up with a
> portable way of testing network services (like ftp clients) without
> setting up a dedicated machine with a well-known name, that
> would be good.  Alternatively, making the name of this
> machine configurable when jtreg is run would also
> be an improvement, and a much simpler one.  But the obvious
> idea of using environment variables doesn't work.  Most environment
> variables are not passed to the running java test program.

Making it configurable, or even ignorable with keywords would be crucial
for distribution testing. Most distributions don't allow their build
daemons to access the network. But for quality control it is essential
that they do run the full test suite.

I haven't made an inventory of what services would be needed on a
machine to be replace javaweb.sfbay.sun.com with a public machine, but
we can certainly run some services on icedtea.classpath.org or maybe one
of the openjdk.java.net machines.

> If it's considered acceptable for IcedTea hackers to get their
> hands dirty with not-100%-free technology, y'all could try
> running the jtreg tests against IcedTea, vanilla OpenJDK7,
> OpenJDK6, and JDK 6u6, and comparing the test failures.

:) Well, the whole idea behind IcedTea is to provide an OpenJDK
derivative that doesn't depend on any non-free build or runtime
requirements.

But I am certainly interested in comparing results. I do think OpenJDK6
and IcedTea are now so close that we shouldn't be seeing any test result
differences between the two.

That brings up the question how to export a JTreport/JTwork environment.
I only posted the text results http://icedtea.classpath.org/~mjw/jtreg/
since the JTreport and JTwork files have all kinds of hard coded
absolute path references. It would be nice to be able to export it all
so I can upload it to some public site for others to look at and compare
with.

Cheers,

MArk