make test TEST="micro:" got java.lang.UnsupportedClassVersionError

[Claes Redestad]

Hi lx,

the need to add --enable-preview is known, and it's a bug introduced a
couple of months ago thanks to a suggestion I did (without realizing
that compiling one micro with --enable-preview would poison all of
them..). This issue is tracked by a few bugs, see
https://bugs.openjdk.java.net/browse/JDK-8250669 and
https://bugs.openjdk.java.net/browse/JDK-8253828

The fix will likely be to remove the --enable-preview from the micros
build after Records goes out of preview and then think more carefully on
how to add support for benchmarking preview features.

As to your other question: When adding a microbenchmark it's appreciated
if the default configuration makes it as quick to run as possible
without sacrificing quality. Often the default number of forks,
iteration runtime et.c. can be tuned down by using custom settings in 
@Fork et.c. Custom parameters used can be limited to a select handful.

Still it's normal for a "good" microbenchmark to require
at least five, usually ten minutes per @Benchmark to produce reasonably
trustworthy results.

I can't speak for what others do, but we (Oracle) select what to run
regularly from the available microbenchmarks at our discretion, though.
That means adding a microbenchmark that takes a prohibitively long time
won't be disruptive to our nightly testing, but for obvious (mostly
economic) reasons we'd be unlikely to add an extremely long-running
micro to regular coverage.

/Claes

On 2020-10-27 10:06, Liu, Xin wrote:
> Hi,
> 
> I follow the instruction on https://htmlpreview.github.io/?https://github.com/openjdk/jdk/blob/master/doc/testing.html#microbenchmarks to run a new microbenchmark.
> I got error message as follows.
> java.lang.UnsupportedClassVersionError: Preview features are not enabled for org/openjdk/bench/java/lang/reflect/proxy/jmh_generated/ProxyBench_newProxyInstance1i_jmhTest (class file version 60.65535). Try running with '--enable-preview'
>          at java.base/java.lang.ClassLoader.defineClass1(Native Method)
>          at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1010)
>          at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
>          at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:855)
>          at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:753)
>          at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:676)
>          at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:634)
>          at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
>          at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519)
>          at java.base/java.lang.Class.forName0(Native Method)
>          at java.base/java.lang.Class.forName(Class.java:377)
>          at org.openjdk.jmh.util.ClassUtils.loadClass(ClassUtils.java:72)
>          at org.openjdk.jmh.runner.BenchmarkHandler.<init>(BenchmarkHandler.java:68)
>          at org.openjdk.jmh.runner.BaseRunner.runBenchmark(BaseRunner.java:232)
>          at org.openjdk.jmh.runner.BaseRunner.doSingle(BaseRunner.java:138)
>          at org.openjdk.jmh.runner.BaseRunner.runBenchmarksForked(BaseRunner.java:75)
>          at org.openjdk.jmh.runner.ForkedRunner.run(ForkedRunner.java:72)
>          at org.openjdk.jmh.runner.ForkedMain.main(ForkedMain.java:84)
> 
> 
> If I add an extra flag “—enable-preview” , the micro bench will work.
> make test TEST="micro:java.lang.reflect" MICRO="VM_OPTIONS=--enable-preview"
> 
> I am using jmh from make/devkit/createJMHBundle.sh, which gives me jmh-core-1.26.jar
> Is that jmh-core too old or we should adjust RunTest.gmk a little bit to have that flag?
> 
> Does Openjdk CI run benchmarks to detect performance regression?  If so,  could you point me to a guideline of microbenmark?
> I lack of the basic sense of a “good” microbenchmark.  My concern is the new micro I added is too long to break the established test run.
> 
> Thanks,
> --lx
> 
>