Early Access Build Test Results

[Jonathan Gibbons]

On 01/11/2013 02:26 PM, Stuart Marks wrote:
> Hi Rory, Balchandra,
>
> It's great to see test results being posted! Now others on the OpenJDK 
> mailing list will know what I'm talking about when I occasionally 
> grumble about test failures. :-)
>
> On 1/11/13 7:13 AM, Balchandra Vaidya wrote:
>>>> Finally, how do the jtreg options used to generate the reported 
>>>> results
>>>> compare to the jtreg options used in the "make test" target?
>>
>> Good question.  Unfortunately, I could not get consistent passes when 
>> I run
>> "make jdk_all" or "make jdk_default" targets.  I ran individual 
>> target (jdk_nio,
>>   jdk_security1, jdk_rmi, ....) separately and then merged the results.
>> However, my chosen targets ran ~3600 tests.
>>
>> Then,  I used jtreg directly to run the tests under following 
>> directories.
>> http://download.java.net/jdk8/testresults/docs/dir.list
>> So, ~4000 tests passed now.
>>
>> The above dir.list do not include awt, 2d and some swing test 
>> directories
>> for which I could not get consistent results.
>>
>> The caveat here (same as choosing the separate make target) is that I 
>> may
>> miss the tests when a new test directory added to the testbase! Any
>> suggestions?
>
> I think that using the Makefile targets is the right starting point.
>
> The Makefiles (e.g., TOP/jdk/test/Makefile) supply the right options 
> to jtreg. They also respect the "problem list" 
> (TOP/jdk/test/ProblemList.txt) which is a list of tests that fail 
> often, but which for a variety of reasons we cannot fix right away 
> (e.g., awaiting a fix from Hotspot, or diagnosing hard-to-find 
> intermittent failures). We want to avoid running these tests because 
> seeing them fail doesn't add any information, and it adds noise. 
> (Eventually, of course, the problem list should shrink down to zero.)
>
> The Makefile targets in turn map to various subdirectories of the 
> tests. Sometimes this mapping is non-obvious, and it's easy to omit 
> tests inadvertently if one is listing subdirectories explicitly, so I 
> think you're wise to be concerned about trying to maintain a lists 
> directories. Development is generally responsible for making sure that 
> the Makefile targets list the right set of directories.
>
> That said, which Makefile targets should you run?
>
> It may be wise to avoid awt, 2d, and swing tests and the like, because 
> they probably require a window system to be running. (They also have 
> interactive tests, but these can be excluded.) Sometimes they'll work 
> if a window system is running on the system, but sometimes not, and 
> then the tests either fail or hang.
>
> There are some properties files in TOP/make/jprt.properties that 
> define "test sets" that we run on our internal build and test systems. 
> Each test set is a set of Makefile targets. Probably the starting 
> point is the "core testset", which basically includes all the tests 
> that test the "headless" parts of the JDK. This includes JVM tests, 
> langtools, jdk_lang, jdk_util, jdk_io, jdk_net, jdk_nio, etc.
>
> Extracting the right list of targets is a bit difficult. You might 
> just have to look at the properties file and copy out the targets of 
> interest. But the targets in the various testsets do change from time 
> to time, so a list you copied could get out of date. We may need to 
> consider adjusting the properties files and the makefiles, since this 
> information is sort of "locked inside" these files and seems hard to 
> use in other contexts.
>
> Also, each Makefile target does a separate jtreg run. This means that 
> each will have separate reports and such that will need to be 
> combined. I don't know how difficult that is.
>
> Anyway, good to see this moving forward.
>
> s'marks

Stuart, Rory,

I suggest there should be a new test/Makefile target for "run all 
recommended tests in a single jtreg run".

-- Jon