Early Access Build Test Results

[Stuart Marks]

Hi Balchandra,

What you've done seems reasonable given that you're just trying to get the 
initial setup going.

I really think it would be preferable to use the actual makefile targets 
instead of assembling a directory list. The reason is that the makefile targets 
not only have lists of directories, they may also specify various jtreg options 
differently depending upon the target. For example, the jdk_rmi tests are all 
run in othervm mode (a run mode of jtreg) which is different from most of the 
other makefile targets. This information is only present in the makefile targets.

Would it be difficult to run all the makefile targets individually and combine 
the results? Yes, there are (I think) 17 makefile targets, but it wouldn't be 
too bad if the combination process could be automated.

My main concern is that if you end up running a different set of tests from our 
internal nightly build and internal developer builds, the results would be 
difficult to compare.

s'marks

On 1/14/13 4:02 AM, Balchandra Vaidya wrote:
> I think you made all valid points. Here are some my observations:
>
> Issue 1: As you described, jdk_all and jdk_default targets depends on individual
> targets and invoke jtreg once. The issue is, the make seems to exit with Error
> when
> there is an error(or failure!) in any individual target! Effectively, I never got
> jdk_all and jdk_default targets completed start to finish.
>
> Issue 2: jdk_all target includes awt and swing tests.  There are some instability
> at the moment and difficult to get consistent results. Some awt tests may required
> to be run on OS console.
>
> Issue 3: jdk_default target do not include targets such as jdk_bean1,
> jdk_bean2,etc.
> and those tests are good and runs without any issue.  That is, using
> jdk_default target
> runs many fewer tests (See Issue 1, however) than we would like to run.
>
> Therefore, I tried
>
> 1) running individual targets (wrapped in a shell script) and merging test results
>   (jtreg -ro). I managed to get to run ~3600 tests.
>
> 2) selected dir.list  and passed it to jtreg directly. I managed to run ~4000
> tests. So,
> started to publish this option.
>
> Obviously, this is not an ideal/clean solution because tests added under a new
> directory or under a new make target may not be executed.  So, centrally managed
> 'dir.list' (include stable directory) might help.