Running tests

[Alan Bateman]

On 26/10/2012 05:51, Magnus Ihse Bursie wrote:
>  From at least my point of view, there has not really been time to think about this, since we are so busy with just getting build-infra to replace the old build. I think there is a lot of good potential here, but I'm not sure what. In any case, I think that discussion needs to wait until build-infra has gotten further along.
>
> The "make test TEST=..." was a first attempt. As you maybe have seen, I just published a WebRev needed to get that to work. (It actually broke in the build-infra forest as well, without anyone noticing...)
>
> I think it could be reasonable to add test dependencies such as jtreg to configure. The only hard thing is how to act when non-essential (non-build) dependencies are missing. Should we abort configure? That would force everyone to have all non-essential tools. Should we ignore it and generate an incomplete configuration? That would make us fail at make time instead, violating the configure principle that a finished configure call should guarantee no missing build dependencies. This question recently crept up with some specific additional closed-source build tools, but it is relevant to test tools just as well.
>
> /Magnus
>
I understand the priority is to get the new build completed and do the 
big switch. However I think it's important that we have some idea how 
tests might be run too.

On the test harness then I could imagine doing "configure 
--with-jtreg=<path-to-jtreg>" and if the location of jtreg is not 
configured then I guess "make test" could fail with a message to 
indicate that jtreg has not been configured.

In a local development environment I could imagine, at a high level, 
doing "make test" and it would build the JDK and run the tests. I 
realize there may be several variants being built and this raises the 
question as to what JDK image is used. There is also the question as to 
whether to test with a non-images build or whether tests should only be 
run on images builds.

If I understand what you have today then the test targets are passed via 
TEST=...  I think we will need to continue with specify test targets as 
most of the time we want to run tests for specific areas.  This also 
helps with partitioning the test execution across machines, which is 
what we do in environments like JPRT in Oracle. There are cross cutting 
targets that might make sense, so called "smoke testing" for example but 
that would require broader discussion and tagging of tests for this to work.

In a test environment that uses a pre-existing build then it would be 
important to be able to specify the JDK to use. It's not clear to me 
whether this fits with configure or not, probably not. Such usage would 
also be useful in the local development environment too, say wanting to 
run the tests on a personal build and the same tests on an 
official/released build to compare results.

I also think concurrency is worth discussion. To be honest, I'm a bit 
uneasy with --with-num-cores at configure time because the parallelism 
is something that I like to decide at the time I run "make". I might 
want to use all resources when doing my initial build, and change to 
cores=1 after that. The existing jdk/test/Makefile is already setup to 
support concurrency and all of the core libraries areas tests can run in 
this mode now, we're just not exploiting it yet. If there is work done 
to determine the right value of num-cores for builds then this value may 
be a candidate for running tests too, at least in build + test farms, 
perhaps not local environments.

Anyway, I think that the ability to run the tests is important and time 
should be given to it once we are over the current hump.

-Alan