Proposal to revise forest graph and integration practices for JDK 9

[Joe Darcy]

Hi Vladimir,

On 11/23/2013 11:01 AM, Vladimir Kozlov wrote:
> Hi Joe,
>
> Please, clarify Hotspot forest. From your description I understand 
> that we will have the next graph where we will do sync up and down 
> between Dev and HotSpot forests:
>
> jdk9 master <- jdk9 dev <-> jdk9 HotSpot
>
> What about HotSpot groups forests? Will they be separate as now or we 
> will have only one forest like HotSpot-main now?

For the time being, the HotSpot group forests will remain. However, with 
improvements to testabilty of HotSpot, over time I think the group 
forests could be retired.

Thanks,

-Joe

>
> Thanks,
> Vladimir
>
> On 11/23/13 8:34 AM, Joe Darcy wrote:
>> Hello,
>>
>> The current arrangements of sets of integration forests for a JDK 
>> platform release, like JDK 8, impose high overheads on
>> development. I'm proposing we use an alternate forest arrangement for 
>> JDK 9 that will dramatically reduce the
>> propagation time of fixes across the set of forests. More details below.
>>
>> JDK release projects for new Java SE platforms, like JDK 8 for Java 
>> SE 8, have long used a graph of forests structured
>> roughly as follow:
>>
>> * A master forest for the release
>>
>> * A thicket of integration forests for particular teams or technology 
>> areas. Today in JDK 8, the "TL" forest hosts
>> changes in tools related to javac as well as core libraries. Forests 
>> for various client libraries (2D, awt, swing) host
>> changes in those areas. A HotSpot forest, fed in turn by several 
>> HotSpot team forests, hosts VM-related changes.
>> Generally, each integration forest only accepts changes to a subset 
>> of repositories. For example, the HotSpot-related
>> forests typically only accept changes to the hotspot repository. The 
>> TL forest accepts changes to library-related repos
>> (jdk, jaxp, jaxws, etc.) and langtools, but not to hotspot.
>>
>> After some amount of testing and other verification steps, changes in 
>> one integration forest are integrated into master,
>> typically on a weekly or bi-weekly basis. From master, the fix then 
>> propagates down to integration forests according to
>> the policies of that forest. A fix could be in master for several 
>> days or more before being propagated to a particular
>> integration forest.
>>
>> While this structure has provided a great deal of cross-team 
>> isolation, it has come at the cost of high propagation
>> delays of fixes to all forests. This propagation delay combined with 
>> only pushing fixes to a subset of repos also
>> severely complicates making coordinated changes which span across 
>> repositories, as often occurred in Project Lambda and
>> which chronically occurs for technologies like servicability. To give 
>> a representative example, consider a hypothetical
>> change which requires updates to both the HotSpot runtime area as 
>> well as core libraries. If the change is first pushed
>> to HotSpot, the propagation proceeds like:
>>
>>      fix pushed to HotSpot runtime forest -> HotSpot main -> JDK 8 
>> master -> TL -> core libs engineer's forest
>>
>> At this point, the libraries half of the change can be pushed:
>>
>>      core libs fix pushed to TL -> JDK 8 master -> HotSpot main -> 
>> HotSpot runtime -> HotSpot runtime engineer's forest
>>
>> This cycle to separately push both halves of what is conceptually a 
>> single fix and wait for them to propagate can take
>> about four weeks. (Worse, often there needs to be a third push to 
>> complete the fix since the first push is often to
>> "accept old way or new way" and the third push updates this to 
>> "accept new way only." When such a clean-up push is
>> needed, it takes another two weeks to fully propagate.)
>>
>> Since more projects requiring cross-repository coordination are 
>> expected in the future, I'm proposing a number of
>> changes to the forest structure and management policies in JDK 9 to 
>> reduce the propagation delays.
>>
>> * A master forest that is a time-delayed version of dev; dev 
>> described below.
>>
>> * The dev forest conceptually replaces TL and hosts all 
>> libraries-related changes. When the sources in dev are in a
>> known-good state, that state can be integrated to master. This 
>> integration cycle would happen at least weekly. In a
>> change from current practice, HotSpot changes would be integrated 
>> into dev and *not* into master. All other team forests
>> would also integrate into dev rather than master.
>>
>> * Coordinated HotSpot + other component fixes would *both* get first 
>> pushed through the HotSpot forest. From hotspot the
>> full fix would be integrated into dev. If additional testing was 
>> appropriate for the non-HotSpot fix, that testing
>> should occur before integration into dev.
>>
>> * Regular promoted builds based on master would continue.
>>
>> By having team forests integrate directly into dev as well as having 
>> many libraries developers pushing directly to dev,
>> the dev forest serves as an active collaboration area with greatly 
>> reduced propagation times across the whole system.
>> With this model there is less cross-team isolation; teams and 
>> individuals are responsible for promptly fixing any
>> breakage which is introduced. If problems are not quickly addressed, 
>> a problematic changeset may be anti-delta'ed.
>>
>> (Conceptually, in this model a separate master forest is not strictly 
>> needed since the known-good states could be
>> indicated using a mechanism like Hg tags. However, while adjusting to 
>> the new model and to allow for fixes directly to
>> master in exceptional circumstances, I'm proposing a physically 
>> separate master forest be retained. The distinct URL of
>> master will also clearly indicate known-good states of the source code.)
>>
>> Please send comments on the above by November 29.
>>
>> Thanks,
>>
>> -Joe
>>