Preparation of update releases

[Rob McKenna]

On 24/10/18 09:15, Volker Simonis wrote:
> On Wed, Oct 24, 2018 at 8:55 AM Rob McKenna <rob.mckenna at oracle.com> wrote:
> >
> >
> >
> > > On 23 Oct 2018, at 19:15, Rob McKenna <rob.mckenna at oracle.com> wrote:
> > >
> > >> On 23/10/18 19:41, Volker Simonis wrote:
> > >>> On Tue, Oct 23, 2018 at 6:23 PM Rob McKenna <rob.mckenna at oracle.com> wrote:
> > >>>
> > >>>> On 23/10/18 17:45, Volker Simonis wrote:
> > >>>>> On Fri, Oct 19, 2018 at 6:19 PM Rob McKenna <rob.mckenna at oracle.com> wrote:
> > >>>>>
> > >>>>> 7 too many!
> > >>>>>
> > >>>>> So, with the new bar for approvals in the jdk-updates project, this
> > >>>>> should become less of an issue. Which is to say, a bug that can be
> > >>>>> pushed via the open repo should be.
> > >>>>>
> > >>>>
> > >>>> Yes, but this doesn't solve the problem of duplicate changes and the
> > >>>> inability of the community to follow which non-security changes will
> > >>>> be in the next security update if such a changes get manually
> > >>>> "cherry-picked" from the open updates repo into the closed security
> > >>>> updates repository.
> > >>>
> > >>> Prior to RDP2 all open contributions will end up in the corresponding
> > >>> update release.
> > >>>
> > >>
> > >> What do you mean here? Prior to RDP2 of 11.0.1 all changes to
> > >> jdk/jdk11 and jdk-updates/jdk11u will end up in jdk-11.0.1 and later?
> > >> When exactly was RDP2 of 11.0.1?
> > >
> > > jdk/jdk11 is the JDK11 GA feature release. All changes in this repo
> > > should be in JDK 11.
> > >
> > > jdk-updates/jdk11u is the updates-release repo. Any changes pushed here
> > > prior to the RDP2 date of an update release will make it into that
> > > update release.
> > >
> > > The RDP2 date for 11.0.1 was in late July. (you'll note this was prior
> > > to the creation of the openjdk jdk11u repo, which was delayed while we
> > > figured a few things out with the release model - hopefully that delay
> > > will be avoided in future)
> > >
> > >
> > >>
> > >>> Kevin gave a jql query below which should help though, is that
> > >>> insufficient?
> > >>>
> > >>
> > >> I get more and more comfortable with it :)
> > >>
> > >> The problem is that I still see a lot of changes in 11.0.1 (e.g. for
> > >> "8204667: Resources not freed on exception") which have not been
> > >> communicated on the vuln-dev mailing list. At the same time, I can't
> > >> look at their JBS issues. So I wonder if they are considered "security
> > >> fixes" but not communicated on vuln-dev (which would be bad) or if
> > >> their JBS issues are just not visible. But in the latter case (i.e. if
> > >> they are not security fixes) shouldn't they appear on Kevin's query?
> > >> "8204667" for example isn't visible there.
> > >
> > > This does appear on Kevin's list, but I don't believe it needs to be
> > > confidential.
> >
> > Sorry, it doesn’t appear because it’s confidential presumably. I’m not sure we can avoid this situation completely. (there may be non-vuln fixes that depend on a vuln fix for example)
> 
> But that's exactly the problem! I'll bring this up on vuln-dev as well
> (which is a closed list unfortunately, so I can't CC it here). If such
> changes can't be opened but at the same time they aren't
> communicated/distributed on the vuln-dev list, there's no chance for
> down-stream builders/packagers to prepare and test a security release
> which is equivalent to the up-stream one. And this is true even if the
> down-stream builders/packagers are members of the vulnerability group.
> 
> Any idea how this problem could be solved?

Not in the context of jdk-updates-dev, for obvious reasons. I'm thinking:

1) we need to ensure that these issues are opened, public and pushed via
the open repo where possible.

2) non-vuln closed issues may need to be distributed along with the vulns
on vuln-dev.

    -Rob

> 
> Thanks,
> Volker
> 
> >
> >     -Rob
> >
> > >
> > >    -Rob
> > >
> > >
> > >>
> > >>>    -Rob
> > >>>
> > >>>>
> > >>>> That's why I was suggesting the creation of an "open" clone for every
> > >>>> security update repository in order to transparently collect the
> > >>>> non-security changes there. This still doesn't solve the issue with
> > >>>> "duplicate" changes after merging the security-repo back into the main
> > >>>> updates repo, but it makes it clear for everybody which non-security
> > >>>> changes will be in an security update.
> > >>>>
> > >>>>>    -Rob
> > >>>>>
> > >>>>>> On 19/10/18 07:43, Kevin Rushforth wrote:
> > >>>>>> Correction, there are 7 new public fixes in 11.0.1.
> > >>>>>>
> > >>>>>> -- Kevin
> > >>>>>>
> > >>>>>>
> > >>>>>>> On 10/19/2018 7:35 AM, Kevin Rushforth wrote:
> > >>>>>>> I'm sure Rob will respond to the rest of this, but here is a correct
> > >>>>>>> filter to find all bugs that are newly fixed in 11.0.1:
> > >>>>>>>
> > >>>>>>> https://bugs.openjdk.java.net/issues/?jql=project%20%3D%20JDK%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%2011.0.1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20not%20in%20(hgupdate-sync))
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> There are a only 8 new public fixes that went into 11.0.1. The
> > >>>>>>> "hgupdate-sync" label indicates a fix synced in from an earlier release
> > >>>>>>> in the same release family, so excluding those will give you an accurate
> > >>>>>>> picture.
> > >>>>>>>
> > >>>>>>> -- Kevin
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On 10/19/2018 7:07 AM, Volker Simonis wrote:
> > >>>>>>>> Hi,
> > >>>>>>>>
> > >>>>>>>> after 11.0.1 has been successfully released I'd like to describe some
> > >>>>>>>> of my observations on how this release has been prepared and suggest
> > >>>>>>>> some improvements to the process:
> > >>>>>>>>
> > >>>>>>>> - I first, naively expected that 11.0.1 will only contain security
> > >>>>>>>> fixes (i.e. the fixes circulated and discussed on the vuln-dev mailing
> > >>>>>>>> list)
> > >>>>>>>> - in the end I was a little surprised that in addition to the ~20
> > >>>>>>>> security fixes 11.0.1 also contained ~130 other changes
> > >>>>>>>> - so in the end 11.0.1 is not strictly speaking a "security release"
> > >>>>>>>> but more a kind of combined "security" and "maintenance" release.
> > >>>>>>>> - because 11.0.1 was prepared in a hidden clone inside Oracle, it is
> > >>>>>>>> hard for others to understand which of the changes in jdk11u will also
> > >>>>>>>> be integrated into 11.0.1. (I know I can list all the issues fixed in
> > >>>>>>>> 11.0.1 in JBS, but this gives me more than 1000 changes which is not
> > >>>>>>>> near the additional ~130 changes which are in 11.0.1 compared to 11).
> > >>>>>>>> - in the end, this makes it hard for anybody outside Oracle to prepare
> > >>>>>>>> and test a security update like 11.0.1, which will be close to what
> > >>>>>>>> will be finally released in the OpenJDK updates project, even if he
> > >>>>>>>> has access to the required security changes (because he simply can not
> > >>>>>>>> know which other changes Oracle integrates into the corresponding
> > >>>>>>>> security update).
> > >>>>>>>>
> > >>>>>>>> I would therefore like to propose the following procedure for the
> > >>>>>>>> creation of future security updates:
> > >>>>>>>>
> > >>>>>>>> - for every security update, create a corresponding "xxx-open" clone
> > >>>>>>>> in the OpenJDK updates project. E.g. for 11.0.1 we could have created
> > >>>>>>>> "jdk-updates/jdk11u-11.0.1-open"
> > >>>>>>>> - the hidden repository required for the collection of security
> > >>>>>>>> changes should be cloned from the corresponding "-open" repo
> > >>>>>>>> - the hidden repository should only be used for pushing security fixes
> > >>>>>>>> - all other fixes intended for that respective security release should
> > >>>>>>>> be downported from the corresponding general updates repository (e.g.
> > >>>>>>>> jdk11u) where they have to be downported first, into the corresponding
> > >>>>>>>> "-open" repository (e.g. jdk11u-11-0.1-open).
> > >>>>>>>> - the hidden repository should regularly merge in the changes from the
> > >>>>>>>> corresponding "-open" repository.
> > >>>>>>>>
> > >>>>>>>> The benefits of this approach would be:
> > >>>>>>>>  - it is transparent for everybody what non-security changes will be
> > >>>>>>>> in the next release
> > >>>>>>>>  - for parties having access to the security fixes it would be trivial
> > >>>>>>>> to prepare and test an update release behind closed walls which will
> > >>>>>>>> exactly correspond to what the final OpenJDK update will be.
> > >>>>>>>>
> > >>>>>>>> The only downside I can see so far is:
> > >>>>>>>>  - the extra effort of creating the new "-open" clone in the updates
> > >>>>>>>> project (but notice that such a clone will in general only live for
> > >>>>>>>> about three month after which he can be switched to read-only mode or
> > >>>>>>>> even deleted after the corresponding hidden repository was merged in
> > >>>>>>>> the general updates repo.)
> > >>>>>>>>
> > >>>>>>>> What do you think? I'm of course especially interested in the opinions
> > >>>>>>>> of Oracle and potential future Update Project maintainers.
> > >>>>>>>>
> > >>>>>>>> Thank you and best regards,
> > >>>>>>>> Volker
> > >>>>>>>
> > >>>>>>
> >