Why do we need a bunch of duplicate changes in jdk7u/hotspot ?

[Erik Trimble]

I don't see any reasonable way to avoid this "problem".

Basically, the problem boils down to this:

when an HS version is forked off for stabilization into a Update 
release, there exists a set of bugs (to be found) that exist in that 
forked HS codebase (let's call that HSa). Naturally, those bugs exist in 
the ongoing 7u-dev HS codebase as well (let's call that HSb).

Now, when one of those bugs is found and fixed in HSb as important for 
the stabilization release, there's a bunch of possibilities:

     (1) immediately sync the HSa repository into HSb
     (2) delay the HSa -> HSb sync until HSa is completed for release
     (3) transplant the fix from HSa to HSb (resulting in the same 
BugID, but a different ChangesetID)
     (4) create a whole new BugID for the fix, and push that changeset 
into HSb
     (5) recognize that current (mainline) work is making the fix 
irrelevant, and thus, we don't want it at all in HSb

Option 1 and 2 produce problems with the situation put forth in Option 
5, as a full repo sync from HSa -> HSb may introduce fixes that are no 
longer desirable in the mainline development repo, so, frankly, they're 
not something that we should do.

Option 3 combined with a general HSa -> HSb sync at the end of the HSa 
stabilization release breaks a rule we've set up for ourselves, in that 
repositories should NOT multiple changesets with the same BugID. This of 
course is merely a guideline, and we can break it at the community's desire.

Option 4 is a bit messy, since it requires that people recognize that 
two different BugIDs actually address the exact same code flaw. It gets 
messier, since when doing a HSa -> HSb sync at the end of a HSa 
stabilization period, whomever is doing the sync has to actually look at 
the code for each potential BugID fix being sync'd back into HSb, to see 
if that BugID is already fixed in the HSa source base.  Normally, if 
this isn't done, it's no big deal, since it merely means that there are 
two changesets that fix the exact same code portion.  However, in the 
case where we recognized that we actually DON'T want to fix the Bug in 
the HSb mainline code, it requires a bit of extra vigilance to make sure 
it doesn't slip back in from the HSa release.


There were very significant issues around merging back changes from 7u2 
into 7u-dev Hostpot, mostly due to the unusual way 7u1 Hotspot was 
managed.  In the case of 7u4, the decision to push in a hs23 version 
from mailine, rather than use the hs22 version from 7u3, caused 
additional grief.


Honestly, the only way to avoid these issue is to do the following:

(a) after a fork of Hotspot for a stabilization release, declare that 
there will NEVER be an actual repository sync from that fork back into 
the mainline development tree, OR vice versa.  This is hard to do, as 
circumstances always interfere with the best intentions.

(b) require all developers who fix a bug in the stabilization release to 
separately push into the mainline repository; however the developer much 
check that the bugfix is still applicable to the mainline repo (i.e. 
check that other development hasn't obsoleted the fix, or that the fix 
doesn't break something else).


In theory the above should be simple, but like everything else in life, 
the devil is in the details. It creates a whole bunch of extra work (vs 
bulk, end-of-stabilization re-integrations) for both developers and QA 
people, and also requires a bit of crystal-ball-gazing around the 
applicability of a fix, whereas bulk re-integrations are a bit more 
brutal in application (i.e. in a bulk integration, the general rule is: 
if its still broke in the mainline, re-integrate the fix from the 
stabilization release even if there might be a future redesign 
obsoleting the fix).


There is no silver bullet.

:-(

-Erik


On 4/27/2012 3:23 PM, David Holmes wrote:
> Hi Volker,
>
> On 27/04/2012 11:15 PM, Volker Simonis wrote:
>> Hi David,
>>
>> thank you very much for the detailed explanation and the links.
>> I missed that discussion (there are just too many mailing lists to
>> monitor) and now it's obviously too late to complain:)
>>
>> But as far as I understood, Eriks mail  "Community Input on Hotspot 22
>> integration into 7u"
>> (http://mail.openjdk.java.net/pipermail/jdk7u-dev/2011-August/000161.html) 
>>
>> was about problems integrating hs22 into 7u.
>>
>> Did these same problems also occur for hs23 as well? I assume yes and
>> if so your explanation would sound perfectly clear to me.
>
> I assume that the same "merge" based model has been followed since 
> that point.
>
>> On the other hand I saw no definitive answer to Edvard's mail "
>> Proposal on how we should handle syncing between stabilisation forests
>> and always open 7u"
>> (http://mail.openjdk.java.net/pipermail/jdk7u-dev/2011-August/000161.html). 
>>
>>
>> Does this mean that we will constantly see these kind of problems
>> (i.e. duplicate changesets with the same bug ID or even worse
>> duplicate changesets with different bug IDs) in the future?
>
> It will occur as long as we are dealing with hsN-1 and hsN and we 
> merge the way we are.
>
>> Is there any "official" document available which explains the syncing
>> between stabilisation forests and 7u?
>
> I'm not aware of anything. I'm also not sure this is specifically an 
> issue with stabilization forests - the 7u-dev forest will have the 
> same issue.
>
> David
> -----
>
>> Thank you and best regards,
>> Volker
>>
>> On Fri, Apr 27, 2012 at 12:45 PM, David 
>> Holmes<david.holmes at oracle.com>  wrote:
>>> Hi Volker,
>>>
>>>
>>> On 27/04/2012 6:59 PM, Volker Simonis wrote:
>>>>
>>>> could anybody please explain why it was necessary to reintegrate a
>>>> whole lot of changes into hs23 of jdk7u/hotspot which have been in the
>>>> hotspot repository since hs22 anyway?
>>>
>>>
>>> This was discussed last year (with thanks to John Rose for these
>>> references):
>>>
>>> http://mail.openjdk.java.net/pipermail/jdk7u-dev/2011-August/000161.html 
>>>
>>> http://mail.openjdk.java.net/pipermail/jdk7u-dev/2011-October/000582.html 
>>>
>>> http://mail.openjdk.java.net/pipermail/jdk7u-dev/2011-November/000748.html 
>>>
>>>
>>> Basically what you are seeing are fixes that went into hs22 and hs23 
>>> after
>>> hs23 forked. hs22 went into 7u2 which became the basis for 7u3 then 
>>> 7u4.
>>> Then hs23 had to be integrated into 7u4 and so you end up with 
>>> different
>>> changesets that duplicate the same actual changes.
>>>
>>> David
>>> -----
>>>
>>>
>>>> This all happened with the following bulk integrations:
>>>>
>>>> http://mail.openjdk.java.net/pipermail/jdk7u-dev/2012-April/002588.html 
>>>>
>>>> http://mail.openjdk.java.net/pipermail/jdk7u-dev/2012-April/002651.html 
>>>>
>>>>
>>>> and it affects (among others):
>>>>
>>>> 7059019: G1: add G1 support to the SA
>>>> 7092412: G1: Some roots not marked during an initial mark that gets an
>>>> evacuation failure
>>>> 7045232: G1: pool names are inconsistent with other collectors (don't
>>>> have 'Space')
>>>> 7068215: G1: Print reference processing time during remark
>>>> 7091032: G1: assert failure when NewRatio is used
>>>> 7092245: G1: Wrong format specifier in G1PrintRegionLivenessInfo 
>>>> header
>>>> output
>>>> 7092238: G1: Uninitialized field gc_efficiency in
>>>> G1PrintRegionLivenessInfo output
>>>> 6484982: G1: process references during evacuation pauses
>>>> 7075646: G1: fix inconsistencies in the monitoring data
>>>> 7086533: G1: assert(!_g1->is_obj_dead(obj)): We should not be
>>>> preserving dead objs: g1CollectedHeap.cpp:3835
>>>> 7097053: G1: assert(da ? referent->is_oop() :
>>>> referent->is_oop_or_null()) failed: referenceProcessor.cpp:1054
>>>> 7097048: G1: extend the G1 SA changes to print per-heap space 
>>>> information
>>>> 7092236: java/util/EnumSet/EnumSetBash.java fails
>>>> 7092278: "jmap -finalizerinfo" throws
>>>> "sun.jvm.hotspot.utilities.AssertionFailure: invalid cp index 0 137"
>>>> 7096366: PPC: corruption of floating-point values with DeoptimizeALot
>>>> 7100165: JSR 292: leftover printing code in methodHandleWalk.cpp
>>>>
>>>> Looking for each of this changes will show you that they are present
>>>> two times in the repository with exactly the same changesets. E.g.:
>>>> http://hg.openjdk.java.net/jdk7u/jdk7u4/hotspot/log?rev=7059019
>>>>
>>>> Regards,
>>>> Volker