Adding Microbenchmarks to the JDK forest/trees (JEP-230)

[Staffan Friberg]

Hi Jon,

On 12/05/2014 01:52 PM, Jonathan Gibbons wrote:
>
> On 12/05/2014 11:46 AM, Staffan Friberg wrote:
>> Hi,
>>
>> On 12/04/2014 10:23 PM, joe darcy wrote:
>>> Hello,
>>>
>>> On 12/4/2014 4:34 PM, David Holmes wrote:
>>>> Hi Staffan,
>>>>
>>>> On 2/12/2014 10:08 AM, Staffan Friberg wrote:
>>>>> Hi,
>>>>>
>>>>> Hopefully this is the right list for this discussion.
>>>>>
>>>>> As part of adding Microbenchmarks to the OpenJDK source tree, I'm 
>>>>> trying
>>>>> to understand how we best would add the benchmark sources to the
>>>>> existing OpenJDK tree structure.
>>>>
>>>> Is there a reason this needs to be inside the OpenJDK instead of a 
>>>> stand-alone project? If it ends up in its own repo and the repo is 
>>>> not needed by anything else, then it is already like a stand-alone 
>>>> project.
>>>
>>> I think David is raising a good question here. A related question is 
>>> if we want to update/fix the microbenchmarks, in how many release 
>>> trains will we want to make that fix? If the expected answer is much 
>>> greater than one, to me that would seem to me to argue for a 
>>> separate forest in the overall OpenJDK effort, but not bundled into 
>>> a particular release.
>>>
>>> For example, in the past, the webrev.ksh script was included in the 
>>> JDK forest. That was an improvement over every engineer having his 
>>> or her personal fork, but still made keeping webrev updated 
>>> unnecessarily difficult since any changes would need to be done 
>>> multiple time and there is nothing fundamentally binding a version 
>>> of webrev to a particular JDK release. Likewise, even though (to my 
>>> knowledge) jtreg is only used for testing the JDK, jtreg has its own 
>>> repository and release schedule separate from any JDK release.
>>>
>>> So if the microbenchmarks are to a first-approximation version 
>>> agnostic, then they should probably have a forest not associated 
>>> with a JDK release. If they are tightly bound to a release, that 
>>> argues for putting them into the JDK forest itself.
>>>
>> The reasons are similar to co-locating functional tests, stable test 
>> base that is not moving after FC, tests support new features and all 
>> work with the JDK you are developing.
>>
>> Looking at some of the goals in the JEP and why I think co-locating 
>> helps.
>>
>>  * Stable and tuned benchmarks, targeted for continuous performance 
>> testing
>>      o A stable and non-moving suite after the Feature Complete
>>        milestone of a feature release, and for non-feature releases
>>
>> To fulfill this having a separate repository would force us to branch 
>> and create builds in sync with the JDK we want to test. As the number 
>> of JDK version increases so will the microbenchmark suite. Making it 
>> complex for anyone adding or running tests to keep track of which 
>> version one should use for a specific JDK branch.
>>
>> As an example, for a stable branch we probably do not want to update 
>> the JMH version, unless there is a critical bug, as doing so can 
>> invalidate any earlier results causing us to lose all history that is 
>> critical when tracking a release. However during development of a new 
>> release we probably want to do that to take advantage of new features 
>> in JMH. And we might require new features in JMH to support changes 
>> in the JDK.
>>
>> Fixing bugs in microbenchmarks will in a similar fashion to any bugs 
>> sometimes need to be backported, but we would have similar struggles 
>> in a separate repository if we want to keep the suite stable for 
>> multiple releases and have multiple branches to support that.
>>
>> JMH similar to jtreg is a separate repository already, and will stay 
>> that way. The co-located part would be the benchmarks, which similar 
>> to the tests written using jtreg benefit from being co-located.
>>
>>
>>  * Simplicity
>>      o Easy to add new benchmarks
>>      o Easy to update tests as APIs and options change, are deprecated,
>>        or are removed during development
>>      o Easy to find and run a benchmark
>>
>> Having it co-located reduces the steps for anyone developing a new 
>> feature to push benchmarks at the same time as the feature in the 
>> same way as functional tests can be pushed.
>>
>> Any benchmark relying on a feature being removed can easily be 
>> deleted without concern about reducing the test coverage of older 
>> releases still providing it.
>>
>> Having it in a completely separate repository I fear will cause 
>> microbenchmark to be out of sight and out of mind, which is the 
>> opposite of the intention of creating the suite. We want more people 
>> to write and run benchmarks in a simple way. Co-location will allow 
>> anyone to simply just add and do make to build for their current 
>> branch and test the feature they are working on, and those benchmarks 
>> will directly be picked up for regression testing once pushed.
>>
>>
>>
>> Hope this helps to further clarify the reason for having the suite as 
>> part of the OpenJDK source tree.
>>
>> Regards,
>> Staffan
>
> Staffan,
>
> You make a good point for it being within the OpenJDK source tree. But 
> you can still use a separate repository in the forest, so that it is 
> versioned, tracked, copied along with the other trees in the forest.
>
> -- Jon

Agree, and I believe we have reached that conclusion as the best option 
in the "other part" of the thread.

I liked the idea of starting of in the top repository as Magnus 
suggested, and move out if required, but as Mark confirmed the historic 
diffs would still be around and keeping the top repository focused on 
Makefiles and other meta data I think make sense as well.

Regards,
Staffan