Early Access Build Test Results

[Stuart Marks]

Hi Rory, Balchandra,

It's great to see test results being posted! Now others on the OpenJDK mailing 
list will know what I'm talking about when I occasionally grumble about test 
failures. :-)

On 1/11/13 7:13 AM, Balchandra Vaidya wrote:
>>> Finally, how do the jtreg options used to generate the reported results
>>> compare to the jtreg options used in the "make test" target?
>
> Good question.  Unfortunately, I could not get consistent passes when I run
> "make jdk_all" or "make jdk_default" targets.  I ran individual target (jdk_nio,
>   jdk_security1, jdk_rmi, ....) separately and then merged the results.
> However, my chosen targets ran ~3600 tests.
>
> Then,  I used jtreg directly to run the tests under following directories.
> http://download.java.net/jdk8/testresults/docs/dir.list
> So, ~4000 tests passed now.
>
> The above dir.list do not include awt, 2d and some swing test directories
> for which I could not get consistent results.
>
> The caveat here (same as choosing the separate make target) is that I may
> miss the tests when a new test directory added to the testbase!  Any
> suggestions?

I think that using the Makefile targets is the right starting point.

The Makefiles (e.g., TOP/jdk/test/Makefile) supply the right options to jtreg. 
They also respect the "problem list" (TOP/jdk/test/ProblemList.txt) which is a 
list of tests that fail often, but which for a variety of reasons we cannot fix 
right away (e.g., awaiting a fix from Hotspot, or diagnosing hard-to-find 
intermittent failures). We want to avoid running these tests because seeing 
them fail doesn't add any information, and it adds noise. (Eventually, of 
course, the problem list should shrink down to zero.)

The Makefile targets in turn map to various subdirectories of the tests. 
Sometimes this mapping is non-obvious, and it's easy to omit tests 
inadvertently if one is listing subdirectories explicitly, so I think you're 
wise to be concerned about trying to maintain a lists directories. Development 
is generally responsible for making sure that the Makefile targets list the 
right set of directories.

That said, which Makefile targets should you run?

It may be wise to avoid awt, 2d, and swing tests and the like, because they 
probably require a window system to be running. (They also have interactive 
tests, but these can be excluded.) Sometimes they'll work if a window system is 
running on the system, but sometimes not, and then the tests either fail or hang.

There are some properties files in TOP/make/jprt.properties that define "test 
sets" that we run on our internal build and test systems. Each test set is a 
set of Makefile targets. Probably the starting point is the "core testset", 
which basically includes all the tests that test the "headless" parts of the 
JDK. This includes JVM tests, langtools, jdk_lang, jdk_util, jdk_io, jdk_net, 
jdk_nio, etc.

Extracting the right list of targets is a bit difficult. You might just have to 
look at the properties file and copy out the targets of interest. But the 
targets in the various testsets do change from time to time, so a list you 
copied could get out of date. We may need to consider adjusting the properties 
files and the makefiles, since this information is sort of "locked inside" 
these files and seems hard to use in other contexts.

Also, each Makefile target does a separate jtreg run. This means that each will 
have separate reports and such that will need to be combined. I don't know how 
difficult that is.

Anyway, good to see this moving forward.

s'marks