Early Access Build Test Results

[Balchandra Vaidya]

On 01/16/13 02:58 AM, Stuart Marks wrote:
> Jon,
>
> I (somewhat) agree! :-)
>
> The current state of affairs is as you say, merely an artifact of our 
> internal build system. It would indeed be preferable for Balchandra to 
> gather up the right set of tests and run them in a single run of jtreg.
>
> The problem is that there is some information encoded in the makefile 
> targets that would be lost if that were done. It's not just the lists 
> of directories. Some of the targets run the tests in othervm mode, and 
> some in agentvm mode. Some of the targets call a 
> SharedLibraryPermissions macro that does some magic. (Hm, see big 
> comment at line 670 of jdk/test/Makefile.) The makefile targets also 
> invoke jtreg with a specific set of options, including using the 
> problem list as the exclude list.

I believe make targets get run during jprt build process and then RE will
create a full build. After that, I take RE build and run tests. So, I  
think,
running tests with slightly different option is not necessarily a problem -
rather I would think it confirm stability/quality of the tests!  
Moreover, at
least in theory, jtreg option itself should not alter test output 
(except test
failure with resource/logistical issues such as memory, timeout,  etc.. ).


Assuming jtreg options are managed to avoid logistical issues, any failure
probably indicates requirement of manual intervention to check whether
any improvement possibility in test or product.


>
> I'd like to avoid having this information extracted from the makefiles 
> and copied into external scripts somewhere. They'll eventually 
> diverge, resulting in confusion.
>
> Ideally there would be some makefile hackery to emit this information 
> in a form directly usable by outside scripts, but that will take some 
> time.
>
> Maybe some of the information could be extracted manually and placed 
> into the repo itself, in a fashion that external scripts can use. This 
> is kind of like the duplication between jprt.properties and the 
> makefiles. This is unpleasant, but at least it confines the 
> duplication to be within the repo.
>
> Meanwhile, in the short term, it might be possible for Balchandra to 
> invoke the 17(!) different makefile targets and merge the results. 

There were 22 (excluding jdk_awt and jdk_swing) targets, and were 
running ~3600 tests combined.
But, I see some changes in Makefile targets since my last try.  I will 
rerun and see how many tests it
will run.



Thanks
Balchandra




> This is also unpleasant, but if it's not too much trouble, it might be 
> reasonable to do. On the other hand, if it turns into a real pain for 
> some reason or another, maybe we needn't bother.
>
> s'marks
>
> On 1/15/13 4:32 PM, Jonathan Gibbons wrote:
>> Uugh uugh uugh.
>>
>> Stuart, I (somewhat) disagree.  The current way of running the tests 
>> is a
>> horrible artifact of our internal JPRT system, and the desire to 
>> split the test
>> load up across machines.   It is not something I would recommend sane 
>> person
>> doing by choice.
>>
>> I agree with the need to have coordination between the way the small 
>> targets
>> work and a bigger target, but the way to do this is to merge the 
>> directories
>> given on the jtreg command line, not try and merge the output of 
>> multiple test
>> runs.
>>
>> -- Jon
>>
>>
>>
>> On 01/15/2013 04:18 PM, Stuart Marks wrote:
>>> Hi Balchandra,
>>>
>>> What you've done seems reasonable given that you're just trying to 
>>> get the
>>> initial setup going.
>>>
>>> I really think it would be preferable to use the actual makefile 
>>> targets
>>> instead of assembling a directory list. The reason is that the makefile
>>> targets not only have lists of directories, they may also specify 
>>> various
>>> jtreg options differently depending upon the target. For example, 
>>> the jdk_rmi
>>> tests are all run in othervm mode (a run mode of jtreg) which is 
>>> different
>>> from most of the other makefile targets. This information is only 
>>> present in
>>> the makefile targets.
>>>
>>> Would it be difficult to run all the makefile targets individually and
>>> combine the results? Yes, there are (I think) 17 makefile targets, 
>>> but it
>>> wouldn't be too bad if the combination process could be automated.
>>>
>>> My main concern is that if you end up running a different set of 
>>> tests from
>>> our internal nightly build and internal developer builds, the 
>>> results would
>>> be difficult to compare.
>>>
>>> s'marks
>>>
>>> On 1/14/13 4:02 AM, Balchandra Vaidya wrote:
>>>> I think you made all valid points. Here are some my observations:
>>>>
>>>> Issue 1: As you described, jdk_all and jdk_default targets depends on
>>>> individual
>>>> targets and invoke jtreg once. The issue is, the make seems to exit 
>>>> with Error
>>>> when
>>>> there is an error(or failure!) in any individual target! 
>>>> Effectively, I
>>>> never got
>>>> jdk_all and jdk_default targets completed start to finish.
>>>>
>>>> Issue 2: jdk_all target includes awt and swing tests.  There are some
>>>> instability
>>>> at the moment and difficult to get consistent results. Some awt 
>>>> tests may
>>>> required
>>>> to be run on OS console.
>>>>
>>>> Issue 3: jdk_default target do not include targets such as jdk_bean1,
>>>> jdk_bean2,etc.
>>>> and those tests are good and runs without any issue.  That is, using
>>>> jdk_default target
>>>> runs many fewer tests (See Issue 1, however) than we would like to 
>>>> run.
>>>>
>>>> Therefore, I tried
>>>>
>>>> 1) running individual targets (wrapped in a shell script) and 
>>>> merging test
>>>> results
>>>>   (jtreg -ro). I managed to get to run ~3600 tests.
>>>>
>>>> 2) selected dir.list  and passed it to jtreg directly. I managed to 
>>>> run ~4000
>>>> tests. So,
>>>> started to publish this option.
>>>>
>>>> Obviously, this is not an ideal/clean solution because tests added 
>>>> under a new
>>>> directory or under a new make target may not be executed.  So, 
>>>> centrally
>>>> managed
>>>> 'dir.list' (include stable directory) might help.
>>>
>>