Early Access Build Test Results

[Stuart Marks]

Jon,

I (somewhat) agree! :-)

The current state of affairs is as you say, merely an artifact of our internal 
build system. It would indeed be preferable for Balchandra to gather up the 
right set of tests and run them in a single run of jtreg.

The problem is that there is some information encoded in the makefile targets 
that would be lost if that were done. It's not just the lists of directories. 
Some of the targets run the tests in othervm mode, and some in agentvm mode. 
Some of the targets call a SharedLibraryPermissions macro that does some magic. 
(Hm, see big comment at line 670 of jdk/test/Makefile.) The makefile targets 
also invoke jtreg with a specific set of options, including using the problem 
list as the exclude list.

I'd like to avoid having this information extracted from the makefiles and 
copied into external scripts somewhere. They'll eventually diverge, resulting 
in confusion.

Ideally there would be some makefile hackery to emit this information in a form 
directly usable by outside scripts, but that will take some time.

Maybe some of the information could be extracted manually and placed into the 
repo itself, in a fashion that external scripts can use. This is kind of like 
the duplication between jprt.properties and the makefiles. This is unpleasant, 
but at least it confines the duplication to be within the repo.

Meanwhile, in the short term, it might be possible for Balchandra to invoke the 
17(!) different makefile targets and merge the results. This is also 
unpleasant, but if it's not too much trouble, it might be reasonable to do. On 
the other hand, if it turns into a real pain for some reason or another, maybe 
we needn't bother.

s'marks

On 1/15/13 4:32 PM, Jonathan Gibbons wrote:
> Uugh uugh uugh.
>
> Stuart, I (somewhat) disagree.  The current way of running the tests is a
> horrible artifact of our internal JPRT system, and the desire to split the test
> load up across machines.   It is not something I would recommend sane person
> doing by choice.
>
> I agree with the need to have coordination between the way the small targets
> work and a bigger target, but the way to do this is to merge the directories
> given on the jtreg command line, not try and merge the output of multiple test
> runs.
>
> -- Jon
>
>
>
> On 01/15/2013 04:18 PM, Stuart Marks wrote:
>> Hi Balchandra,
>>
>> What you've done seems reasonable given that you're just trying to get the
>> initial setup going.
>>
>> I really think it would be preferable to use the actual makefile targets
>> instead of assembling a directory list. The reason is that the makefile
>> targets not only have lists of directories, they may also specify various
>> jtreg options differently depending upon the target. For example, the jdk_rmi
>> tests are all run in othervm mode (a run mode of jtreg) which is different
>> from most of the other makefile targets. This information is only present in
>> the makefile targets.
>>
>> Would it be difficult to run all the makefile targets individually and
>> combine the results? Yes, there are (I think) 17 makefile targets, but it
>> wouldn't be too bad if the combination process could be automated.
>>
>> My main concern is that if you end up running a different set of tests from
>> our internal nightly build and internal developer builds, the results would
>> be difficult to compare.
>>
>> s'marks
>>
>> On 1/14/13 4:02 AM, Balchandra Vaidya wrote:
>>> I think you made all valid points. Here are some my observations:
>>>
>>> Issue 1: As you described, jdk_all and jdk_default targets depends on
>>> individual
>>> targets and invoke jtreg once. The issue is, the make seems to exit with Error
>>> when
>>> there is an error(or failure!) in any individual target! Effectively, I
>>> never got
>>> jdk_all and jdk_default targets completed start to finish.
>>>
>>> Issue 2: jdk_all target includes awt and swing tests.  There are some
>>> instability
>>> at the moment and difficult to get consistent results. Some awt tests may
>>> required
>>> to be run on OS console.
>>>
>>> Issue 3: jdk_default target do not include targets such as jdk_bean1,
>>> jdk_bean2,etc.
>>> and those tests are good and runs without any issue.  That is, using
>>> jdk_default target
>>> runs many fewer tests (See Issue 1, however) than we would like to run.
>>>
>>> Therefore, I tried
>>>
>>> 1) running individual targets (wrapped in a shell script) and merging test
>>> results
>>>   (jtreg -ro). I managed to get to run ~3600 tests.
>>>
>>> 2) selected dir.list  and passed it to jtreg directly. I managed to run ~4000
>>> tests. So,
>>> started to publish this option.
>>>
>>> Obviously, this is not an ideal/clean solution because tests added under a new
>>> directory or under a new make target may not be executed.  So, centrally
>>> managed
>>> 'dir.list' (include stable directory) might help.
>>
>