RFR [XS]: 8248429: Add --enable-preview as VM argument when running microbenchmarks

[Peter Levart]

On 7/7/20 7:51 PM, Roger Riggs wrote:
> Hi,
>
> I have found it very useful to be able to run benchmarks against multiple
> versions of the JDK.  Build the benchmark jar once and to compare 
> results.
> If all of the classes are built with --enable-preview, none of them 
> will run
> against older JDKs. So an approach that only compiles those files that 
> need
> preview would be more useful. 
...
> Or perhaps, separate out preview based benchmarks
> into a separate jar.
>
> $.02, Roger


Perhaps the last option is the way to go. Why? Preview features are 
specific to a major OpenJDK release so while a particular preview 
feature may be present as a preview feature in two consecutive releases, 
you can not run a class with the preview feature compiled for OpenJDK N 
on an OpenJDK N+1 or vice versa. So a single benchmarks.jar file makes 
sense only for benchmarks that don't use preview features and are 
compiled with (or for) lowest release possible...


The benchmarks using preview feature OTOH will need to be re-classified 
once the preview feature graduates and becomes a mainline feature. So 
once it is classified differently in OpenJDK N+1 as it was in OpenJDK N, 
it can not be built with such classification on (or for) the OpenJDK N 
release any more because, among other things, names of packages/modules 
may change. So these two sets of benchmarks (using or not using preview 
features) have different constraints and is therefore reasonable for 
them to be built separately and packed into separate benchmarks.jar files.


Packing to separate .jar files might be a pragmatic solution for a 
problem that Jorn Vernee discovered: the JMH compiler plugin generates 
two files in the output directory which are included in the benchmarks.jar:

/META-INF/BenchmarkList

/META-INF/CompilerHints


While the 1st one is "updated" incrementally if it already exists, its 
modification is not protected by any kind of locking mechanism and so 
concurrent compilation by two or more instances of javac may produce 
garbled result. The 2nd one seems to be overwritten entirely by content 
of a single compile session, so its final form does not represent 
compiler hints for all aggregated benchmarks and may therefore produce 
skewed results for some benchmarks. I see two solutions for that problem:

- fix JMH to correctly handle concurrent incremental updates to both 
above files and; or

- compile the two sets of benchmarks separately with separate output 
directories and create separate benchmarks.jar files for them


I think the 2nd option is a simpler, pragmatic solution.


Peter


>
>
> On 7/7/20 10:26 AM, Peter Levart wrote:
>> I suggest adding --enable-preview to JMH_JVM_ARGS in general now (it 
>> doesn't hurt even if classes are not compiled with --enable-preview) 
>> and then take time to devise an effective strategy for selectively 
>> compiling micro benchmarks with or without --enable-preview. At least 
>> so the benchmarks would work out-of-the-box when run via make test.
>>
>> WDYT?
>>
>>
>> Regards, Peter
>>
>>
>> On 6/30/20 10:15 PM, Claes Redestad wrote:
>>> On 2020-06-30 22:12, Magnus Ihse Bursie wrote:
>>>>>
>>>>> Second to that a solution in the build would be preferable - if we 
>>>>> can
>>>>> come up with something that has infinitesimal impact to build times.
>>>> Are we talking about many files? Could you consider listing those 
>>>> files explicitly in the makefile? That would make it cheap to 
>>>> filter them out from the normal compilation, and instead do a 
>>>> secondary compilation with them.
>>>
>>> Right now there's one micro using --enable-preview, so that'd be a very
>>> short list.
>>>
>>> /Claes
>