[gnu.andrew at redhat.com: Re: OpenJDK 8u and Backport Bugs]

[Severin Gehwolf]

Hi Christoph,

Sorry, previous email escaped unintentionally... (crtl+return pressed
by mistake)

On Tue, 2020-12-15 at 07:44 +0000, Langer, Christoph wrote:
> Hi Andrew,
> 
> sorry for being late on this discussion, I just now found the time to
> go through the mail thread on 8u-dev in detail. But, hold on, why are
> you proposing this change to the process? What issues are you aiming
> to solve?

Note that we've largely discussed this in the context of 8u. But in
general we'd like to have a process that works for most (all?) updates
releases.

The main goal is for us to have a way to distribute backporting work.
The intention is to solve a couple of problems with it

- Allow for proper reporting. Time a backport bug gets opened until
  it is resolved kind of metrics. Using labels for this isn't working.
- Assign specific backport issues to a certain user. This is a more
  formal approach than labels. It has the advantage of getting email
  notifications when certain events happen. Like a backport bug is
  created and assigned to someone. Somebody other than the backporter
  can assign bugs. The assignee would notice this by getting email
  notifications.
- For CSR-needing bugs we need to do the explicit backport bug
  anyway. It would no longer be an exception to the rule.
- Commenting on the master bug that somebody is doing the backport
  isn't really machine readable. Getting a report quickly as to who
  is doing the backport is difficult, especially figuring this out
  for different releases. 

I understand this also has the drawback of an additional step (creating
the explicit bug), but we haven't found an alternative solution to
cover all the cases. If you've got suggestions, I'd be all ears.

> So far, and it has worked very well at least in 11u, the process was
> to add fix request labels in the parent/master bug that is to be
> backported. Also, when somebody wants/needs to claim that they are
> working on a backport, they would claim it in a comment on the master
> bug. Eventually, this comment could also be modified to be the actual
> "Fix Request" comment with more details for the approver.

The theory is that it works well when only few people do the work. It
has a scalability problem. This process is rather implicit and relies
on the process knowledge of the backporter. We feel the explicit bug is
easier to understand for new contributors.

> For most cases, there is no need to create a backport item in advance
> as hgupdater will create it at the moment the backport is pushed.
> Exceptions are when e.g. CSRs have to be done. So, after all, we
> should not encourage to create backport items in advance. This can as
> well lead to orphaned items, e.g. when the person doing the backport
> then steps away and forgets about it or when the target release is
> labeled wrong and hgupdater creates another bug, leaving the intended
> backport alone.

Makes sense. I wonder though, how it's different to the current
process. One could go comment on the bug "I'm doing a backport of this
for OpenJDK X" and then step away. Note that the explicit backport bug
would remain unresolved in that case. Again, it would show up in the
reporting and would get flagged.

> I know about the issues with 8-pool and 11-pool. For 8-pool,
> resolving OpenJDK backport releases doesn't work at all. And for 11-
> pool there's a danger that hgupdater will seize a backport that is
> intended for OpenJDK 11u backports when Oracle does a backport at the
> same time. Both can be circumvented by explicitly specifying the
> intended target update release, such as "openjdk8u291" or "11.0.11".
> In that case, however, there's a certain danger if the backporter
> misses a release and pushes the backport to a different release than
> set at the backport. Then a fresh backport issue would be created and
> the already existing one gets orphaned. So, both approaches have
> potential issues and need specific attention by the committers. Hence
> the safest way is to not open a backport bug at all and let hgupdater
> do the work whenever possible.

We've been testing this in 8u with setting the explicit bug to "Fix
Version" -> openjdk8u292 (current jdk8u-dev). Then the bug would get
properly resolved as you say. What we thought we'd do to avoid mistakes
is for the maintainer to set the version correctly when the bug gets
approved for push. So it wouldn't be the backporters responsibility.
For 11u it's a non-issue as 11-pool works fine. It would sometimes need
coordination with Oracle, but it would be an OK compromise IMO.

> As for the issue of copying all the labels to backport bugs,
> especially those fix request/approval labels that will flood the
> open/unpushed backport lists, that is a known issue. The Skara
> tooling has addressed this already ([0], [1]), so for 13u this isn't
> an issue any longer. And for 11u this will be solved once we go to
> Skara (Another reason for not waiting too long until migrating to
> Skara with 11u ��).

Right. I'm not convinced Skara will be a solution for 8u, though. So
there is that problem.

> So, after all, my opinion is: Please don't do this thing with the
> backport bugs. Not as a general process guideline.

This isn't set in stone. Exploratory at this point. We havent't found a
better way to handle bug assignments, though.

> All the discussions and even the transient "jdk8u-andrew" and "jdk8u-
> needs-review" labels ought to be done on the main item.

I agree on that one. Hence my objection to apply the label on the
backport bug. It's a departure from the general updates process
workflow too. So +1 for keeping the labels on the master bugs.

> I would also hope that we could agree to have a consistent process
> between 11u and 8u that avoids these manual backport bugs.

We are open to suggestions. In our experience the existing process
isn't structured enough to do proper assignment and reporting. Doing
this collaboratively is another goal. Individuals from various
institutions doing the work.

Thanks,
Severin

> Thanks & Best regards
> Christoph
> 
> [0] https://bugs.openjdk.java.net/browse/SKARA-819
> [1]      
> https://github.com/openjdk/skara/commit/4d1bcbac9f97925d68b5f5dbb3d0dbc91bdbb420
> 
> 
> > -----Original Message-----
> > From: jdk-updates-dev <jdk-updates-dev-retn at openjdk.java.net> On
> > Behalf Of Andrew Hughes
> > Sent: Freitag, 11. Dezember 2020 18:03
> > To: jdk-updates-dev at openjdk.java.net
> > Subject: [gnu.andrew at redhat.com: Re: OpenJDK 8u and Backport Bugs]
> > 
> > Forwarding for wider input.
> > 
> > The Oracle text on adding fix request labels specifies to use the
> > parent bug, but this has rarely been an issue in practice because
> > we've not been creating backport bugs ahead of time.
> > 
> > So do people have a preference for using the backport bug or
> > the parent bug for fix requests?
> > 
> > The latter is easier for us maintainers, but we are a small
> > group compared with that of all contributors.
> > 
> > ----- Forwarded message from Andrew Hughes <gnu.andrew at redhat.com>
> > -----
> > 
> > Date: Fri, 11 Dec 2020 15:41:43 +0000
> > From: Andrew Hughes <gnu.andrew at redhat.com>
> > To: Severin Gehwolf <sgehwolf at redhat.com>
> > Subject: Re: OpenJDK 8u and Backport Bugs
> > User-Agent: Mutt/1.10.1 (2018-07-13)
> > 
> > On 11:33 Fri 11 Dec     , Severin Gehwolf wrote:
> > > Hi Andrew,
> > > 
> > > On Fri, 2020-12-11 at 04:41 +0000, Andrew Hughes wrote:
> > > > Hi all,
> > > > 
> > > > We've worked out a way we can create backport bugs for OpenJDK
> > > > 8u
> > > > issues and have them correctly resolved by hgupdater.
> > > > 
> > > > For 8u, '8-pool' doesn't work as '11-pool' does with OpenJDK
> > > > 11u. I
> > > > assume this is because '8-pool' is assumed to refer to the
> > > > Oracle fork
> > > > of 8u, which also use 8u281, 8u291, etc. Sadly, a generic
> > > > 'openjdk8u'
> > > > seems not to work either.
> > > > 
> > > > What does work is using the specific version of 'openjdk8ux'.
> > > > We
> > > > therefore propose the following, and I have updated the wiki to
> > > > correspond with this:
> > > > 
> > > > 1. When work is started on a backport, create a backport bug as
> > > > follows:
> > > >   a. Log in to the OpenJDK bug database and go to the bug you
> > > > want to
> > backport.
> > > >   b. Click More → Create Backport
> > > >   c. Set yourself as the Assignee and change Fix Version to
> > > > 'openjdk8u'.
> > > >   d. On the new bug, clear the inherited 'Affected Versions'
> > > > and labels.
> > > >   e. Set 'Affected Versions' to 'openjdk8u'
> > > >   f. Add any desired labels, such as 'jdk8u-<username>', to the
> > > > bug,
> > > >   where <username> is your OpenJDK username.
> > > > 
> > > > 2. Proceed as usual, but apply the jdk8u-needs-review label and
> > > > make
> > > > approval requests on the backport bug.  This avoids the issue
> > > > where
> > > > labels on the parent issue are cloned to other bugs.
> > > 
> > > While point 2) would avoid the need to remove a couple of
> > > additional
> > > labels when the backport bug is being created, it doesn't really
> > > avoid
> > > the need of "clearing labels" entirely. There are very few bugs
> > > without
> > > labels at all.
> > > 
> > > Also, the master bug serves as the place were all the info is
> > > being
> > > gathered. This is usefull, since the JDK 11 backporting info
> > > would be
> > > on the same bug as any JDK 8 backporting info. Doing certain
> > > labels on
> > > an explicit backport bug breaks this. Adding the label on the
> > > backport
> > > bug also suggests to add the "Fix Request" comment to the
> > > backport bug,
> > > moving further info away from the main bug. With my maintainer
> > > hat on,
> > > it's easier to do approvals by looking at the master bug directly
> > > and
> > > see how decisions panned out for other releases.
> > > 
> > > For those reasons I think we should keep this part as-is: Keep
> > > applying
> > > the jdk8u-fix-request label to the master bug. Clearing 2-ish
> > > labels
> > > when creating the backport bug should be fine. I'd be happy to do
> > > that
> > > if people forget.
> > > 
> > > Besides, the intention would be to create the backport bug as
> > > soon as
> > > somebody starts working on it. At that point, no jdk8u-fix-
> > > request
> > > label should be there anyway and, thus, would only apply if JDK
> > > 11u
> > > adopted this process too. Maybe I'm missing something.
> > > 
> > > Thanks,
> > > Severin
> > > 
> > 
> > Yeah, #2 is after the backport bug has been created. What I'm
> > referring to
> > with "labels on the parent issue are cloned to other bugs" is a
> > backport bug
> > being created for another release. For example, I've seen bugs for
> > Oracle
> > backports appear in our queue because they get jdk8u-fix-request
> > added by
> > the cloning process. Even though they also have jdk8u-fix-yes, they
> > don't
> > match the filter because of the fix version not being an OpenJDK 8u
> > one.
> > 
> > The same would happen if the 13u backport was done after 8u too.
> > Ideally, 8u should be the last, but that doesn't always happen. 7u
> > may
> > want to adopt this process too and that would be an issue there.
> > 
> > I'm aware that it's a bit of a pain when it comes to approving the
> > bug, but that's something that really only affects the three of us
> > acting as maintainers and can be worked around easily by opening
> > the
> > parent bug. I think it's simpler and less confusing for someone
> > working on the bug to have one bug to work with, not having to
> > flick
> > between two. Also, having their own 8u backport bug will hopefully
> > encourage them to make it their own and not worry about adding to a
> > bug shared by many others.
> > 
> > I don't know how much of an issue bug noise is for people who
> > aren't
> > interested in the 8u backport process, but this would reduce it.
> > 
> > So, I'd like some feedback from others before making a decision
> > here.
> > It doesn't seem a good idea to base the decision just on what works
> > best for us as maintainers.
> > 
> > Thanks,
> > --
> > Andrew :)
> > 
> > Senior Free Java Software Engineer
> > OpenJDK Package Owner
> > Red Hat, Inc. (http://www.redhat.com)
> > 
> > PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net)
> > Fingerprint = 5132 579D D154 0ED2 3E04  C5A0 CFDA 0F9B 3596 4222
> > 
> > 
> > 
> > ----- End forwarded message -----
> > 
> > --
> > Andrew :)
> > 
> > Senior Free Java Software Engineer
> > OpenJDK Package Owner
> > Red Hat, Inc. (http://www.redhat.com)
> > 
> > PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net)
> > Fingerprint = 5132 579D D154 0ED2 3E04  C5A0 CFDA 0F9B 3596 4222