Proposal to revise forest graph and integration practices for JDK 9

[Jeremy Manson]

Just out of curiosity, how is "known good" defined?  I assume that the
answer to this is well-known within Oracle.

What we do is have a single source base.  Developers have to make sure that
their changes will keep the source base in a known good state before
committing.  Tests are run after every commit (more or less), and if you
broke something, your change is backed out.

This has the advantage of extreme simplicity, but we have to be very
aggressive about best testing practices.

Jeremy


On Sat, Nov 23, 2013 at 11:26 AM, Robert Field <robert.field at oracle.com>wrote:

> +1
>
> The pain of the current system was felt heavily by the lambda team.
>
> -Robert
>
>
>
>
> On 11/23/13 08:34, Joe Darcy wrote:
>
>> Hello,
>>
>> The current arrangements of sets of integration forests for a JDK
>> platform release, like JDK 8, impose high overheads on development. I'm
>> proposing we use an alternate forest arrangement for JDK 9 that will
>> dramatically reduce the propagation time of fixes across the set of
>> forests. More details below.
>>
>> JDK release projects for new Java SE platforms, like JDK 8 for Java SE 8,
>> have long used a graph of forests structured roughly as follow:
>>
>> * A master forest for the release
>>
>> * A thicket of integration forests for particular teams or technology
>> areas. Today in JDK 8, the "TL" forest hosts changes in tools related to
>> javac as well as core libraries. Forests for various client libraries (2D,
>> awt, swing) host changes in those areas. A HotSpot forest, fed in turn by
>> several HotSpot team forests, hosts VM-related changes. Generally, each
>> integration forest only accepts changes to a subset of repositories. For
>> example, the HotSpot-related forests typically only accept changes to the
>> hotspot repository. The TL forest accepts changes to library-related repos
>> (jdk, jaxp, jaxws, etc.) and langtools, but not to hotspot.
>>
>> After some amount of testing and other verification steps, changes in one
>> integration forest are integrated into master, typically on a weekly or
>> bi-weekly basis. From master, the fix then propagates down to integration
>> forests according to the policies of that forest. A fix could be in master
>> for several days or more before being propagated to a particular
>> integration forest.
>>
>> While this structure has provided a great deal of cross-team isolation,
>> it has come at the cost of high propagation delays of fixes to all forests.
>> This propagation delay combined with only pushing fixes to a subset of
>> repos also severely complicates making coordinated changes which span
>> across repositories, as often occurred in Project Lambda and which
>> chronically occurs for technologies like servicability. To give a
>> representative example, consider a hypothetical change which requires
>> updates to both the HotSpot runtime area as well as core libraries. If the
>> change is first pushed to HotSpot, the propagation proceeds like:
>>
>>     fix pushed to HotSpot runtime forest -> HotSpot main -> JDK 8 master
>> -> TL -> core libs engineer's forest
>>
>> At this point, the libraries half of the change can be pushed:
>>
>>     core libs fix pushed to TL -> JDK 8 master -> HotSpot main -> HotSpot
>> runtime -> HotSpot runtime engineer's forest
>>
>> This cycle to separately push both halves of what is conceptually a
>> single fix and wait for them to propagate can take about four weeks.
>> (Worse, often there needs to be a third push to complete the fix since the
>> first push is often to "accept old way or new way" and the third push
>> updates this to "accept new way only." When such a clean-up push is needed,
>> it takes another two weeks to fully propagate.)
>>
>> Since more projects requiring cross-repository coordination are expected
>> in the future, I'm proposing a number of changes to the forest structure
>> and management policies in JDK 9 to reduce the propagation delays.
>>
>> * A master forest that is a time-delayed version of dev; dev described
>> below.
>>
>> * The dev forest conceptually replaces TL and hosts all libraries-related
>> changes. When the sources in dev are in a known-good state, that state can
>> be integrated to master. This integration cycle would happen at least
>> weekly. In a change from current practice, HotSpot changes would be
>> integrated into dev and *not* into master. All other team forests would
>> also integrate into dev rather than master.
>>
>> * Coordinated HotSpot + other component fixes would *both* get first
>> pushed through the HotSpot forest. From hotspot the full fix would be
>> integrated into dev. If additional testing was appropriate for the
>> non-HotSpot fix, that testing should occur before integration into dev.
>>
>> * Regular promoted builds based on master would continue.
>>
>> By having team forests integrate directly into dev as well as having many
>> libraries developers pushing directly to dev, the dev forest serves as an
>> active collaboration area with greatly reduced propagation times across the
>> whole system. With this model there is less cross-team isolation; teams and
>> individuals are responsible for promptly fixing any breakage which is
>> introduced. If problems are not quickly addressed, a problematic changeset
>> may be anti-delta'ed.
>>
>> (Conceptually, in this model a separate master forest is not strictly
>> needed since the known-good states could be indicated using a mechanism
>> like Hg tags. However, while adjusting to the new model and to allow for
>> fixes directly to master in exceptional circumstances, I'm proposing a
>> physically separate master forest be retained. The distinct URL of master
>> will also clearly indicate known-good states of the source code.)
>>
>> Please send comments on the above by November 29.
>>
>> Thanks,
>>
>> -Joe
>>
>>
>