Adding Microbenchmarks to the JDK forest/trees (JEP-230)

[Staffan Friberg]

Hi Jon,

As part of the initial set of benchmarks we hope to add as part of this 
JEP I'm guessing it will be around 200-300 files. This would grow 
overtime, but I believe we won't see tens of thousands of files, it is 
more likely it will be something like a 1000 files.

//Staffan

On 12/02/2014 02:14 PM, Jonathan Gibbons wrote:
> Staffan,
>
> I would also ask how many files are eventually likely to be involved.
>
> If it's tens of files up to low hundreds, then a top level dir makes 
> sense.
>
> If it's tens of thousands of files, then a separate repo makes more 
> sense.
>
> -- Jon
>
>
> On 12/02/2014 02:08 PM, Staffan Friberg wrote:
>> Hi Chris,
>>
>> Agree, there is no major reason this needs to be a new repository, as 
>> I mentioned in the 3 options below it would work well without it. The 
>> main thing I want to achieve is that the benchmarks are located on 
>> the top level. The suite will contain benchmarks for all parts of the 
>> JDK so having it in either jdk or hotspot doesn't feel like it makes 
>> sense. If people agree on having it as folder in the top level JDK 
>> repository I'm perfectly fine with that.
>>
>> As for building it will most likely not be of the general build 
>> process for building the JDK (do not want to increase the compilation 
>> time for anyone not requiring the benchmark suite). It would have its 
>> own target 'build-microbenchmark' which would depend on 
>> 'exploded-image', but not the reverse.
>>
>> //Staffan
>>
>> On 12/02/2014 01:23 PM, Chris Hegarty wrote:
>>> Staffan,
>>>
>>> Having all the benchmarks located in a single place makes sense to 
>>> me, but this doesn’t necessarily mean that they need their own 
>>> repository, in the forest.
>>>
>>> If I can build, run, and test ( usual development cycle ) without 
>>> any dependency on these benchmarks, or their infrastructure, 
>>> essential working with a partial forest ( without the ‘benchmark’ 
>>> repository ), then I can see the possible value in having a separate 
>>> repository ( so I can skip cloning and updating it ). But, I’m not 
>>> sure if that is a reasonable justification for a new repository, as 
>>> it is probably at odds with your goals, or maybe not?
>>>
>>> -Chris
>>>
>>> On 2 Dec 2014, at 19:53, Staffan Friberg 
>>> <staffan.friberg at oracle.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> (Adding the jdk9-dev list to increase the visibility of the 
>>>> discussion)
>>>>
>>>> With the multiple sub-repository commit mechanism improved I 
>>>> believe this might be less of an issue. JPRT can push JDK and HS 
>>>> changes at together and the same functionality should be possible 
>>>> to use for this as well. I wonder if the test issue earlier was 
>>>> that it was a completely separate repository outside of the JDK 
>>>> forest, and less of an issue when being part of the same forest as 
>>>> the JDK source code. Perhaps someone from SQE can chime?
>>>>
>>>> Otherwise the main reason for having a separate sub-repository on 
>>>> the top level is making it easier to find what benchmarks are 
>>>> available and have a single place to add new once avoid any risk of 
>>>> name duplication. JMH is superb in filtering during execution 
>>>> during runtime so running just a single test or a group of tests is 
>>>> very straight forward and the recommended way, rather than having 
>>>> multiple benchmark JARs. It also makes the build process easier as 
>>>> the building can be done using a single Makefile and a single 
>>>> benchmark JAR (actually two, one for JDK 8 compatible tests and one 
>>>> for JDK 9) that can be picked up by automatic performance testing.
>>>>
>>>> Cheers,
>>>> Staffan
>>>>
>>>> On 12/02/2014 06:48 AM, roger riggs wrote:
>>>>> Hi Staffan,
>>>>>
>>>>> An earlier issue was keeping tests in sync with the code under 
>>>>> test, hence
>>>>> the use of test directories within each repository.
>>>>> I think a structure in which the benchmarks for some function and 
>>>>> the function
>>>>> itself are in the same repository that is easier to understand and 
>>>>> maintain.
>>>>>
>>>>> $.02, Roger
>>>>>
>>>>>
>>>>> On 12/1/2014 7:08 PM, Staffan Friberg wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Hopefully this is the right list for this discussion.
>>>>>>
>>>>>> As part of adding Microbenchmarks to the OpenJDK source tree, I'm 
>>>>>> trying to understand how we best would add the benchmark sources 
>>>>>> to the existing OpenJDK tree structure.
>>>>>>
>>>>>> Since the microbenchmark suite will cover all parts of the JDK, 
>>>>>> covering HotSpot, JDK libraries and Nashorn, it would be 
>>>>>> preferred to add the microbenchmark directory as a new top level 
>>>>>> directory. Something similar to the following structure. Having 
>>>>>> "benchmark" as the top-level directory would allow us to later 
>>>>>> add different types of benchmarks without colliding with the 
>>>>>> microbenchmark suite.
>>>>>>
>>>>>> <openjdk-root>/
>>>>>>    benchmark/microbenchmark/...
>>>>>>    hotspot/...
>>>>>>    jdk/...
>>>>>>    nashorn/...
>>>>>>
>>>>>> With this as the premise I can see the following 3 options for 
>>>>>> how this could be added to the source code layout
>>>>>>
>>>>>> 1. Part of jdk-root repository
>>>>>>      * Only makes sense if we want to move in a direction with fewer
>>>>>>        trees (and eventually a single tree)
>>>>>> 2. Part of another already existing tree
>>>>>>      * Not sure if this is possible without converting and moving 
>>>>>> the
>>>>>>        directory to a subdirectory of that tree
>>>>>> 3. New tree in the forest/tree structure
>>>>>>      * Most logical option as it follows the current setup and 
>>>>>> structure
>>>>>>
>>>>>>
>>>>>> Anyone have any comments and/or concerns on the suggested 
>>>>>> directory location and the tree structure in option 3.
>>>>>>
>>>>>> Would the build-dev team be the right group to later help setup a 
>>>>>> new tree if decided to be the right way to go?
>>>>>>
>>>>>> Regards,
>>>>>> Staffan
>>>>>>
>>
>