RFR: 8294142: make test should report only on executed tests [v2]

[Erik Joelsson]

On Fri, 23 Sep 2022 17:34:00 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> This should help to speed up tests significantly. Currently, if we run "make test" with a subset of tests, JTReg would still read the entirety of test root to report on tests that were not run. Even with current suite of tests it gets expensive. If you add more tests to suite -- for example, mounting a large test archive like pre-generated fuzzer tests -- it starts to take a lot of time.
>> 
>> We might default to `-report:executed`. My own CI -- that has millions of tests mounted in `test/` -- was running with `JTREG="OPTIONS=-report:executed"` for years now. I think that should be the default way to run the tests in JDK tree.
>> 
>> There is a wrinkle with jtreg retries: we need to collate together the reports for successful retries from multiple runs. Fortunately, we can then fall back to `-report:all-executed`. [CODETOOLS-7903323](https://bugs.openjdk.org/browse/CODETOOLS-7903323) is poised to provide a new reporting option, `-report:files`, which would make this whole business easier, we can upgrade to it once minimum jtreg version is bumped. This RFE provides faster tests today, and is backportable to update releases.
>> 
>> Motivational example:
>> 
>> 
>> $ time CONF=linux-x86_64-server-release make run-test TEST=gc/epsilon/TestHelloWorld.java
>> 
>> # Default JDK tree
>> 
>> # Baseline
>> real	0m8.813s
>> user	0m20.846s
>> sys	0m3.841s
>> 
>> # Patched
>> real	0m5.926s  ; about 1.5x faster
>> user	0m14.637s
>> sys	0m3.451s
>> 
>> 
>> # +100K fuzzer tests in tree
>> 
>> # Baseline
>> real	3m22.978s
>> user	3m33.417s
>> sys	0m11.197s
>> 
>> # Patched
>> real	0m8.415s   ; about 23x faster
>> user	0m15.623s
>> sys	0m4.746s
>
> Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains three commits:
> 
>  - JTREG REPORT option to be used as default
>  - Merge branch 'master' into JDK-8294142-make-test-executed
>  - Fix

make/RunTests.gmk line 48:

> 46: 
> 47: $(eval $(call ParseKeywordVariable, TEST_OPTS, \
> 48:     SINGLE_KEYWORDS := JOBS TIMEOUT_FACTOR JCOV JCOV_DIFF_CHANGESET REPORT, \

I think we only add options that really are common to more than one test framework to TEST_OPTS.

-------------

PR: https://git.openjdk.org/jdk/pull/10405