Adding Microbenchmarks to the JDK forest/trees (JEP-230)

[Jonathan Gibbons]

Staffan,

I would also ask how many files are eventually likely to be involved.

If it's tens of files up to low hundreds, then a top level dir makes sense.

If it's tens of thousands of files, then a separate repo makes more sense.

-- Jon


On 12/02/2014 02:08 PM, Staffan Friberg wrote:
> Hi Chris,
>
> Agree, there is no major reason this needs to be a new repository, as 
> I mentioned in the 3 options below it would work well without it. The 
> main thing I want to achieve is that the benchmarks are located on the 
> top level. The suite will contain benchmarks for all parts of the JDK 
> so having it in either jdk or hotspot doesn't feel like it makes 
> sense. If people agree on having it as folder in the top level JDK 
> repository I'm perfectly fine with that.
>
> As for building it will most likely not be of the general build 
> process for building the JDK (do not want to increase the compilation 
> time for anyone not requiring the benchmark suite). It would have its 
> own target 'build-microbenchmark' which would depend on 
> 'exploded-image', but not the reverse.
>
> //Staffan
>
> On 12/02/2014 01:23 PM, Chris Hegarty wrote:
>> Staffan,
>>
>> Having all the benchmarks located in a single place makes sense to 
>> me, but this doesn’t necessarily mean that they need their own 
>> repository, in the forest.
>>
>> If I can build, run, and test ( usual development cycle ) without any 
>> dependency on these benchmarks, or their infrastructure, essential 
>> working with a partial forest ( without the ‘benchmark’ repository ), 
>> then I can see the possible value in having a separate repository ( 
>> so I can skip cloning and updating it ). But, I’m not sure if that is 
>> a reasonable justification for a new repository, as it is probably at 
>> odds with your goals, or maybe not?
>>
>> -Chris
>>
>> On 2 Dec 2014, at 19:53, Staffan Friberg <staffan.friberg at oracle.com> 
>> wrote:
>>
>>> Hi,
>>>
>>> (Adding the jdk9-dev list to increase the visibility of the discussion)
>>>
>>> With the multiple sub-repository commit mechanism improved I believe 
>>> this might be less of an issue. JPRT can push JDK and HS changes at 
>>> together and the same functionality should be possible to use for 
>>> this as well. I wonder if the test issue earlier was that it was a 
>>> completely separate repository outside of the JDK forest, and less 
>>> of an issue when being part of the same forest as the JDK source 
>>> code. Perhaps someone from SQE can chime?
>>>
>>> Otherwise the main reason for having a separate sub-repository on 
>>> the top level is making it easier to find what benchmarks are 
>>> available and have a single place to add new once avoid any risk of 
>>> name duplication. JMH is superb in filtering during execution during 
>>> runtime so running just a single test or a group of tests is very 
>>> straight forward and the recommended way, rather than having 
>>> multiple benchmark JARs. It also makes the build process easier as 
>>> the building can be done using a single Makefile and a single 
>>> benchmark JAR (actually two, one for JDK 8 compatible tests and one 
>>> for JDK 9) that can be picked up by automatic performance testing.
>>>
>>> Cheers,
>>> Staffan
>>>
>>> On 12/02/2014 06:48 AM, roger riggs wrote:
>>>> Hi Staffan,
>>>>
>>>> An earlier issue was keeping tests in sync with the code under 
>>>> test, hence
>>>> the use of test directories within each repository.
>>>> I think a structure in which the benchmarks for some function and 
>>>> the function
>>>> itself are in the same repository that is easier to understand and 
>>>> maintain.
>>>>
>>>> $.02, Roger
>>>>
>>>>
>>>> On 12/1/2014 7:08 PM, Staffan Friberg wrote:
>>>>> Hi,
>>>>>
>>>>> Hopefully this is the right list for this discussion.
>>>>>
>>>>> As part of adding Microbenchmarks to the OpenJDK source tree, I'm 
>>>>> trying to understand how we best would add the benchmark sources 
>>>>> to the existing OpenJDK tree structure.
>>>>>
>>>>> Since the microbenchmark suite will cover all parts of the JDK, 
>>>>> covering HotSpot, JDK libraries and Nashorn, it would be preferred 
>>>>> to add the microbenchmark directory as a new top level directory. 
>>>>> Something similar to the following structure. Having "benchmark" 
>>>>> as the top-level directory would allow us to later add different 
>>>>> types of benchmarks without colliding with the microbenchmark suite.
>>>>>
>>>>> <openjdk-root>/
>>>>>    benchmark/microbenchmark/...
>>>>>    hotspot/...
>>>>>    jdk/...
>>>>>    nashorn/...
>>>>>
>>>>> With this as the premise I can see the following 3 options for how 
>>>>> this could be added to the source code layout
>>>>>
>>>>> 1. Part of jdk-root repository
>>>>>      * Only makes sense if we want to move in a direction with fewer
>>>>>        trees (and eventually a single tree)
>>>>> 2. Part of another already existing tree
>>>>>      * Not sure if this is possible without converting and moving the
>>>>>        directory to a subdirectory of that tree
>>>>> 3. New tree in the forest/tree structure
>>>>>      * Most logical option as it follows the current setup and 
>>>>> structure
>>>>>
>>>>>
>>>>> Anyone have any comments and/or concerns on the suggested 
>>>>> directory location and the tree structure in option 3.
>>>>>
>>>>> Would the build-dev team be the right group to later help setup a 
>>>>> new tree if decided to be the right way to go?
>>>>>
>>>>> Regards,
>>>>> Staffan
>>>>>
>