Proposal to revise forest graph and integration practices for JDK 9

[Robert Field]

+1

The pain of the current system was felt heavily by the lambda team.

-Robert



On 11/23/13 08:34, Joe Darcy wrote:
> Hello,
>
> The current arrangements of sets of integration forests for a JDK 
> platform release, like JDK 8, impose high overheads on development. 
> I'm proposing we use an alternate forest arrangement for JDK 9 that 
> will dramatically reduce the propagation time of fixes across the set 
> of forests. More details below.
>
> JDK release projects for new Java SE platforms, like JDK 8 for Java SE 
> 8, have long used a graph of forests structured roughly as follow:
>
> * A master forest for the release
>
> * A thicket of integration forests for particular teams or technology 
> areas. Today in JDK 8, the "TL" forest hosts changes in tools related 
> to javac as well as core libraries. Forests for various client 
> libraries (2D, awt, swing) host changes in those areas. A HotSpot 
> forest, fed in turn by several HotSpot team forests, hosts VM-related 
> changes. Generally, each integration forest only accepts changes to a 
> subset of repositories. For example, the HotSpot-related forests 
> typically only accept changes to the hotspot repository. The TL forest 
> accepts changes to library-related repos (jdk, jaxp, jaxws, etc.) and 
> langtools, but not to hotspot.
>
> After some amount of testing and other verification steps, changes in 
> one integration forest are integrated into master, typically on a 
> weekly or bi-weekly basis. From master, the fix then propagates down 
> to integration forests according to the policies of that forest. A fix 
> could be in master for several days or more before being propagated to 
> a particular integration forest.
>
> While this structure has provided a great deal of cross-team 
> isolation, it has come at the cost of high propagation delays of fixes 
> to all forests. This propagation delay combined with only pushing 
> fixes to a subset of repos also severely complicates making 
> coordinated changes which span across repositories, as often occurred 
> in Project Lambda and which chronically occurs for technologies like 
> servicability. To give a representative example, consider a 
> hypothetical change which requires updates to both the HotSpot runtime 
> area as well as core libraries. If the change is first pushed to 
> HotSpot, the propagation proceeds like:
>
>     fix pushed to HotSpot runtime forest -> HotSpot main -> JDK 8 
> master -> TL -> core libs engineer's forest
>
> At this point, the libraries half of the change can be pushed:
>
>     core libs fix pushed to TL -> JDK 8 master -> HotSpot main -> 
> HotSpot runtime -> HotSpot runtime engineer's forest
>
> This cycle to separately push both halves of what is conceptually a 
> single fix and wait for them to propagate can take about four weeks. 
> (Worse, often there needs to be a third push to complete the fix since 
> the first push is often to "accept old way or new way" and the third 
> push updates this to "accept new way only." When such a clean-up push 
> is needed, it takes another two weeks to fully propagate.)
>
> Since more projects requiring cross-repository coordination are 
> expected in the future, I'm proposing a number of changes to the 
> forest structure and management policies in JDK 9 to reduce the 
> propagation delays.
>
> * A master forest that is a time-delayed version of dev; dev described 
> below.
>
> * The dev forest conceptually replaces TL and hosts all 
> libraries-related changes. When the sources in dev are in a known-good 
> state, that state can be integrated to master. This integration cycle 
> would happen at least weekly. In a change from current practice, 
> HotSpot changes would be integrated into dev and *not* into master. 
> All other team forests would also integrate into dev rather than master.
>
> * Coordinated HotSpot + other component fixes would *both* get first 
> pushed through the HotSpot forest. From hotspot the full fix would be 
> integrated into dev. If additional testing was appropriate for the 
> non-HotSpot fix, that testing should occur before integration into dev.
>
> * Regular promoted builds based on master would continue.
>
> By having team forests integrate directly into dev as well as having 
> many libraries developers pushing directly to dev, the dev forest 
> serves as an active collaboration area with greatly reduced 
> propagation times across the whole system. With this model there is 
> less cross-team isolation; teams and individuals are responsible for 
> promptly fixing any breakage which is introduced. If problems are not 
> quickly addressed, a problematic changeset may be anti-delta'ed.
>
> (Conceptually, in this model a separate master forest is not strictly 
> needed since the known-good states could be indicated using a 
> mechanism like Hg tags. However, while adjusting to the new model and 
> to allow for fixes directly to master in exceptional circumstances, 
> I'm proposing a physically separate master forest be retained. The 
> distinct URL of master will also clearly indicate known-good states of 
> the source code.)
>
> Please send comments on the above by November 29.
>
> Thanks,
>
> -Joe
>