RFR: 8294403: [REDO] make test should report only on executed tests
Aleksey Shipilev
shade at openjdk.org
Tue Jan 3 09:48:42 UTC 2023
This should help to speed up tests significantly. Currently, if we run "make test" with a subset of tests, JTReg would still read the entirety of test root to report on tests that were not run. Even with current suite of tests it gets expensive. If you add more tests to suite -- for example, mounting a large test archive like pre-generated fuzzer tests -- it starts to take a lot of time.
The reasonable default for older jtreg is `-report:executed`. My own CI -- that has millions of tests mounted in `test/` -- was running with `JTREG="OPTIONS=-report:executed"` for years now. I think that should be the default way to run the tests in JDK tree.
[CODETOOLS-7903323](https://bugs.openjdk.org/browse/CODETOOLS-7903323) provides a new reporting option, `-report:files`, which makes this whole business easier. This requires new jtreg.
Motivational improvements:
$ time CONF=linux-x86_64-server-release make run-test TEST=gc/epsilon/TestHelloWorld.java
# Default JDK tree
# Baseline
real 0m9.086s
user 0m22.857s
sys 0m4.202s
# Patched
real 0m6.406s ; +40% faster
user 0m17.156s
sys 0m3.193s
# +100K fuzzer tests in tree
# Baseline
real 3m1.997s
user 3m16.643s
sys 0m10.490s
# Patched
real 0m8.919s ; 20x faster
user 0m17.904s
sys 0m4.860s
Additional testing:
- [x] Ad-hoc timing tests (see above)
- [x] Reproducer from [CODETOOLS-7903331](https://bugs.openjdk.org/browse/CODETOOLS-7903331)
- [x] Eyeballing the test results on passing tests with REPEAT_COUNT > 1
- [x] Eyeballing the test results on intermittently failing tests with RETRY_COUNT > 1
-------------
Commit messages:
- Fix
Changes: https://git.openjdk.org/jdk/pull/11824/files
Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=11824&range=00
Issue: https://bugs.openjdk.org/browse/JDK-8294403
Stats: 12 lines in 3 files changed: 11 ins; 0 del; 1 mod
Patch: https://git.openjdk.org/jdk/pull/11824.diff
Fetch: git fetch https://git.openjdk.org/jdk pull/11824/head:pull/11824
PR: https://git.openjdk.org/jdk/pull/11824
More information about the build-dev
mailing list