Early Access Build Test Results

[Jonathan Gibbons]

Uugh uugh uugh.

Stuart, I (somewhat) disagree.  The current way of running the tests is 
a horrible artifact of our internal JPRT system, and the desire to split 
the test load up across machines.   It is not something I would 
recommend sane person doing by choice.

I agree with the need to have coordination between the way the small 
targets work and a bigger target, but the way to do this is to merge the 
directories given on the jtreg command line, not try and merge the 
output of multiple test runs.

-- Jon



On 01/15/2013 04:18 PM, Stuart Marks wrote:
> Hi Balchandra,
>
> What you've done seems reasonable given that you're just trying to get 
> the initial setup going.
>
> I really think it would be preferable to use the actual makefile 
> targets instead of assembling a directory list. The reason is that the 
> makefile targets not only have lists of directories, they may also 
> specify various jtreg options differently depending upon the target. 
> For example, the jdk_rmi tests are all run in othervm mode (a run mode 
> of jtreg) which is different from most of the other makefile targets. 
> This information is only present in the makefile targets.
>
> Would it be difficult to run all the makefile targets individually and 
> combine the results? Yes, there are (I think) 17 makefile targets, but 
> it wouldn't be too bad if the combination process could be automated.
>
> My main concern is that if you end up running a different set of tests 
> from our internal nightly build and internal developer builds, the 
> results would be difficult to compare.
>
> s'marks
>
> On 1/14/13 4:02 AM, Balchandra Vaidya wrote:
>> I think you made all valid points. Here are some my observations:
>>
>> Issue 1: As you described, jdk_all and jdk_default targets depends on 
>> individual
>> targets and invoke jtreg once. The issue is, the make seems to exit 
>> with Error
>> when
>> there is an error(or failure!) in any individual target! Effectively, 
>> I never got
>> jdk_all and jdk_default targets completed start to finish.
>>
>> Issue 2: jdk_all target includes awt and swing tests.  There are some 
>> instability
>> at the moment and difficult to get consistent results. Some awt tests 
>> may required
>> to be run on OS console.
>>
>> Issue 3: jdk_default target do not include targets such as jdk_bean1,
>> jdk_bean2,etc.
>> and those tests are good and runs without any issue.  That is, using
>> jdk_default target
>> runs many fewer tests (See Issue 1, however) than we would like to run.
>>
>> Therefore, I tried
>>
>> 1) running individual targets (wrapped in a shell script) and merging 
>> test results
>>   (jtreg -ro). I managed to get to run ~3600 tests.
>>
>> 2) selected dir.list  and passed it to jtreg directly. I managed to 
>> run ~4000
>> tests. So,
>> started to publish this option.
>>
>> Obviously, this is not an ideal/clean solution because tests added 
>> under a new
>> directory or under a new make target may not be executed.  So, 
>> centrally managed
>> 'dir.list' (include stable directory) might help.
>