Proposal to revise forest graph and integration practices for JDK 9

[Bengt Rutisson]

On 2013-11-25 02:28, David Holmes wrote:
> On 24/11/2013 5:08 AM, Dmitry Samersoff wrote:
>> Joe,
>>
>> It's unclean for me whether we would keep hotspot-runtime, hotspot-comp,
>> etc forests.
>>
>> I hope not.
>>
>> Besides limited jprt push capability and the fact everybody used to it I
>> see no benefits of hotspot group forests.
>
> There have been many benefits to the isolation provided by the use of 
> the hotspot group forests. The primary one is that isolation.

Yes, I agree that isolation is the primary benefit. However, there are 
also many drawbacks with the group repos for HotSpot. The primary 
drawback is that it often happens that a bug gets introduced in one 
group repo but escapes the testing for that repo. It then takes at least 
a week before the testing in another group repo finds the bug and then 
it takes at least another week before the fix has propagated.

 From my perspective the drawbacks heavily outweigh the benefits of the 
group repos.

> There have been times when a group has been working on a feature that 
> needs to be shared within the group but is not ready for general use. 
> In that case they can hold-off integrating into hotspot-main until 
> they are ready to share with the rest of hotspot.

For large changes, such as JFR or PermGen removal, we have been using 
project repos instead. That is a much better solution in my opinion. For 
semi-large features I think we are now in a situation where we can 
request ad-hoc runs in Aurora reasonably well. So, I think that testing 
should be done before the push instead.

>
> And unless someone magically resolves the test resourcing limitations 
> we also benefit by being able to target nightly testing for the 
> different groups.

I don't fully understand this comment. We can still keep all the nightly 
testing we have right now. We can remove the "main nightly" that we have 
now but keep all the group nightlies. They just need to pick the same 
binary instead different ones. We can also review the overlap between 
the group nightlies and remove any tests that will be run multiple times 
in the same way with the exact same binary.

So, the resources required for testing should be less and we get the 
same testing as we do now. But we get quick feedback when we break 
something and fixes get propagated to everyone quickly.

I think it would be great if we take this chance to get rid of the group 
repos.

Thanks,
Bengt

>
> Let's not throw out the baby with the bath water here.
>
> David
>
>> -Dmitry
>>
>>
>> On 2013-11-23 20:34, Joe Darcy wrote:
>>> Hello,
>>>
>>> The current arrangements of sets of integration forests for a JDK
>>> platform release, like JDK 8, impose high overheads on development. I'm
>>> proposing we use an alternate forest arrangement for JDK 9 that will
>>> dramatically reduce the propagation time of fixes across the set of
>>> forests. More details below.
>>>
>>> JDK release projects for new Java SE platforms, like JDK 8 for Java SE
>>> 8, have long used a graph of forests structured roughly as follow:
>>>
>>> * A master forest for the release
>>>
>>> * A thicket of integration forests for particular teams or technology
>>> areas. Today in JDK 8, the "TL" forest hosts changes in tools 
>>> related to
>>> javac as well as core libraries. Forests for various client libraries
>>> (2D, awt, swing) host changes in those areas. A HotSpot forest, fed in
>>> turn by several HotSpot team forests, hosts VM-related changes.
>>> Generally, each integration forest only accepts changes to a subset of
>>> repositories. For example, the HotSpot-related forests typically only
>>> accept changes to the hotspot repository. The TL forest accepts changes
>>> to library-related repos (jdk, jaxp, jaxws, etc.) and langtools, but 
>>> not
>>> to hotspot.
>>>
>>> After some amount of testing and other verification steps, changes in
>>> one integration forest are integrated into master, typically on a 
>>> weekly
>>> or bi-weekly basis. From master, the fix then propagates down to
>>> integration forests according to the policies of that forest. A fix
>>> could be in master for several days or more before being propagated 
>>> to a
>>> particular integration forest.
>>>
>>> While this structure has provided a great deal of cross-team isolation,
>>> it has come at the cost of high propagation delays of fixes to all
>>> forests. This propagation delay combined with only pushing fixes to a
>>> subset of repos also severely complicates making coordinated changes
>>> which span across repositories, as often occurred in Project Lambda and
>>> which chronically occurs for technologies like servicability. To give a
>>> representative example, consider a hypothetical change which requires
>>> updates to both the HotSpot runtime area as well as core libraries. If
>>> the change is first pushed to HotSpot, the propagation proceeds like:
>>>
>>>      fix pushed to HotSpot runtime forest -> HotSpot main -> JDK 8 
>>> master
>>> -> TL -> core libs engineer's forest
>>>
>>> At this point, the libraries half of the change can be pushed:
>>>
>>>      core libs fix pushed to TL -> JDK 8 master -> HotSpot main ->
>>> HotSpot runtime -> HotSpot runtime engineer's forest
>>>
>>> This cycle to separately push both halves of what is conceptually a
>>> single fix and wait for them to propagate can take about four weeks.
>>> (Worse, often there needs to be a third push to complete the fix since
>>> the first push is often to "accept old way or new way" and the third
>>> push updates this to "accept new way only." When such a clean-up 
>>> push is
>>> needed, it takes another two weeks to fully propagate.)
>>>
>>> Since more projects requiring cross-repository coordination are 
>>> expected
>>> in the future, I'm proposing a number of changes to the forest 
>>> structure
>>> and management policies in JDK 9 to reduce the propagation delays.
>>>
>>> * A master forest that is a time-delayed version of dev; dev described
>>> below.
>>>
>>> * The dev forest conceptually replaces TL and hosts all
>>> libraries-related changes. When the sources in dev are in a known-good
>>> state, that state can be integrated to master. This integration cycle
>>> would happen at least weekly. In a change from current practice, 
>>> HotSpot
>>> changes would be integrated into dev and *not* into master. All other
>>> team forests would also integrate into dev rather than master.
>>>
>>> * Coordinated HotSpot + other component fixes would *both* get first
>>> pushed through the HotSpot forest. From hotspot the full fix would be
>>> integrated into dev. If additional testing was appropriate for the
>>> non-HotSpot fix, that testing should occur before integration into dev.
>>>
>>> * Regular promoted builds based on master would continue.
>>>
>>> By having team forests integrate directly into dev as well as having
>>> many libraries developers pushing directly to dev, the dev forest 
>>> serves
>>> as an active collaboration area with greatly reduced propagation times
>>> across the whole system. With this model there is less cross-team
>>> isolation; teams and individuals are responsible for promptly fixing 
>>> any
>>> breakage which is introduced. If problems are not quickly addressed, a
>>> problematic changeset may be anti-delta'ed.
>>>
>>> (Conceptually, in this model a separate master forest is not strictly
>>> needed since the known-good states could be indicated using a mechanism
>>> like Hg tags. However, while adjusting to the new model and to allow 
>>> for
>>> fixes directly to master in exceptional circumstances, I'm proposing a
>>> physically separate master forest be retained. The distinct URL of
>>> master will also clearly indicate known-good states of the source 
>>> code.)
>>>
>>> Please send comments on the above by November 29.
>>>
>>> Thanks,
>>>
>>> -Joe
>>>
>>
>>