Early Access Build Test Results

[Stuart Marks]

On 1/16/13 3:48 AM, Balchandra Vaidya wrote:
> I believe make targets get run during jprt build process and then RE will
> create a full build. After that, I take RE build and run tests. So, I think,
> running tests with slightly different option is not necessarily a problem -
> rather I would think it confirm stability/quality of the tests! Moreover, at
> least in theory, jtreg option itself should not alter test output (except test
> failure with resource/logistical issues such as memory, timeout,  etc.. ).
>
> Assuming jtreg options are managed to avoid logistical issues, any failure
> probably indicates requirement of manual intervention to check whether
> any improvement possibility in test or product.

It's true, most jtreg options won't affect success or failure of the tests, 
they mostly affect things like reporting and output. Some things are 
significant though.

The agentvm/othervm difference could cause some tests to fail, but as Alan 
pointed out in another message, we should extract that logic from the makefiles 
and move it elsewhere, probably TEST.ROOT.

Looking through the makefile (see the stuff regarding JTREG_BASIC_OPTIONS) 
there are a number of other significant options:

     -a -ea -esa -ignore:quiet -timeoutFactor:4 -J-Xmx512m

Additional args may get passed in via JAVA_ARGS and JAVA_VM_ARGS, but I'm not sure.

It's not necessarily a fatal problem if your test runs using a different set of 
options. But I don't think it necessarily helps confirm the stability or 
quality of the tests, either. The thing I'm concerned about is somebody 
investigating a test that fails in your test run but not on his desktop or in 
our nightly build. That failure might simply have been caused by running the 
test in a different mode.

The good news is that these options don't change very often, so they're 
unlikely to get out of date.

>> Meanwhile, in the short term, it might be possible for Balchandra to invoke
>> the 17(!) different makefile targets and merge the results.
>
> There were 22 (excluding jdk_awt and jdk_swing) targets, and were running ~3600
> tests combined.
> But, I see some changes in Makefile targets since my last try.  I will rerun
> and see how many tests it
> will run.

Ah, the other five targets are jdk_beans[123], jdk_jdi, and jdk_sound. Those 
aren't included in the "default" testset. I'm surprised you got only 3,600 
tests. Our runs of the default testset (the 17 targets) includes 3,900 tests.

s'marks