Early Access Build Test Results

[Balchandra Vaidya]

On 01/14/13 07:36 AM, Stuart Marks wrote:
> On 1/11/13 2:54 PM, Jonathan Gibbons wrote:
>> I suggest there should be a new test/Makefile target for "run all 
>> recommended
>> tests in a single jtreg run".
>
> I think this would be ideal. Implicitly, then, Balchandra's script 
> would just invoke this makefile target, as would other internal 
> build/test systems. This would eliminate copying of information about 
> these targets out of the Makefiles into external scripts, where 
> they'll inevitably get out of date.
>
> I say this is ideal, but this is probably more difficult to achieve 
> than one might think. Not impossible, but probably somewhat tedious.
>
> The "test sets" are defined redundantly in *two* properties files, 
> TOP/make/jprt.properties and TOP/jdk/make/jprt.properties. They are 
> mostly the same, though there are some small differences. I'm not sure 
> if that's intentional. Note also that these aren't plain properties 
> files; they use some kind of variable interpolation and string 
> substitution syntax I'm not familiar with.
>
> Then, there are the makefiles. Oh, the makefiles. Mainly, they are 
> TOP/test/Makefile, TOP/jdk/test/Makefile, and 
> TOP/langtools/test/Makefile. (I don't know how the JVM tests are 
> invoked; probably TOP/hotspot/test/Makefile.)
>
> It looks to me like each of the Makefiles defines several individual 
> test targets, each of which invokes a run of jtreg. Most of these are 
> in the jdk repo. There seem to be targets that invoke "all" of the 
> tests (but not test sets), but I think the "all tests" target just 
> depends on the individual targets, so it'll still invoke jtreg once 
> for each individual target.
>
> Some cleaning up and rearrangement is called for here. There is some 
> redundancy, but it would be preferable to have all the redundancy 
> within files in the forest (as painful as it is) than to have 
> redundant information copied into external scripts.
>
> In short, it's a mess.


I think you made all valid points. Here are some my observations:


Issue 1: As you described, jdk_all and jdk_default targets depends on 
individual
targets and invoke jtreg once. The issue is, the make seems to exit with 
Error when
there is an error(or failure!) in any individual target! Effectively, I 
never got
jdk_all and jdk_default targets completed start to finish.


Issue 2: jdk_all target includes awt and swing tests.  There are some 
instability
at the moment and difficult to get consistent results. Some awt tests 
may required
to be run on OS console.


Issue 3: jdk_default target do not include targets such as jdk_bean1, 
jdk_bean2,etc.
and those tests are good and runs without any issue.  That is, using 
jdk_default target
runs many fewer tests (See Issue 1, however) than we would like to run.


Therefore, I tried

1) running individual targets (wrapped in a shell script) and merging 
test results
  (jtreg -ro). I managed to get to run ~3600 tests.


2) selected dir.list  and passed it to jtreg directly. I managed to run 
~4000 tests. So,
started to publish this option.


Obviously, this is not an ideal/clean solution because tests added under 
a new
directory or under a new make target may not be executed.  So, centrally 
managed
'dir.list' (include stable directory) might help.


Thanks
Balchandra



>
> s'marks