Announcing the Renaissance benchmark suite

[Francois Farquet]

Le 13.05.19 à 11:54, Florian Weimer a écrit :
> * Alan Bateman:
>
>> On 09/05/2019 19:04, Francois Farquet wrote:
>>> Hi Alan,
>>>
>>> The suite has been developed and tested under JDK8 and full support
>>> of JDK11 is planned for the 1.0.0 release.
>>> Regarding the latest JDK and ea versions, we’ll do our best to test
>>> it regularly but if the Java Product Group at Oracle could regularly
>>> check its compatibility and performance, that would be great.
>>>
>>  From what I can tell, none of the apache-spark benchmarks will run
>> with with any recent feature releases. They all fall over with
>> HADOOP-14586 in code that is parsing the value of "java.version"
>> property. The database benches seems to have issues too, seems to be
>> code in one of Chronicle libraries trying to hack
>> java.lang.reflect.AccessibleObject privates. Are you, or others,
>> working with these projects to get fixes or versions that run on more
>> recent releases, maybe as part of the effort to get it to run on JDK
>> 11 builds?
> This leads to the question to what extent these higher-level run-time
> environments are part of the benchmark.  Is it expected they remain
> unchanged (except for such bug fixes)?  Or can they be enhanced to use
> current and new JVM features that help with their implementation?

Each benchmark should be made as compatible as possible with all 
reasonable JDK versions.

However, that doesn't mean there shouldn't be benchmarks that are 
compatible only with some late JDK version if there is a good reason for 
that, like the use of recent features that are representative of what a 
recent workload looks like. So, considering Renaissance as an evolving 
benchmark suite with benchmarks being added or removed from a release to 
another, we can imagine having some JDK version restriction per 
benchmark if we face that need.

Regarding enhancing current benchmarks, I would advise against anything 
different than bug fixes. Major changes should be made as a distinct 
benchmark to avoid confusion.

Thanks,

Francois

>
> Thanks,
> Florian