Parallelizing the full compaction of CMS

[Jon Masamitsu]

On 6/3/2015 1:00 PM, Jeremy Manson wrote:
> I'm not on hotspot-gc-dev, so I'm not seeing the responses.  But I 
> looked online...
>
> Jon says that "My understanding was that the change was based on a 
> parallelization of the serial mark-sweep-compact (used by CMS for full 
> collections). If it was based on the parallel-old compaction used by 
> ParallelGC, then my bad.  It should have been reviewed."
>
> His original understanding was correct - we parallelized the serial 
> mark-sweep-compact used by CMS.  I'm sorry if I misled anyone there.

No problem.  It gave me a chance to say again that
I'd rather not have to support two implementations of
a parallel full stop-the-world collector.  We have that
issue with the two implementations of a parallel young
collector and I'd really rather not go there again.

>
> Otherwise, thanks for your thoughtful reply, Volker.  In my experience 
> of open source projects, if you can't get a maintainer to agree to 
> consider your patch, then there is really very little that you can 
> do.  And I totally get that maintainers don't want / have the time for 
> the burden of understanding huge amounts of complicated code they 
> didn't write!
>
> So, the question comes down to what "community supported" means in 
> this case.  We can't actually submit anything to Hotspot without 
> approval from someone within Oracle.  We have a vested interest in 
> keeping this patch working, but, for major JDK releases, we really 
> only start stress testing to identify problems with it when we upgrade 
> to the latest versions of the JDK, which usually happens some months 
> after Oracle releases it.  That could cause some issues for upstream.
>
> The upside is that I think we'd be willing to do what it takes, if 
> getting it upstreamed is a possibility.  This patch is a huge 
> nuisance, and it is much simpler not to have to do a week or two's 
> worth of forward porting every year.  And whatever problems arise are 
> problems that we have to fix anyway.
>
> Jon, if you are particularly worried, note that we can keep it hidden 
> behind a flag without too much difficulty (and did for a year or two).

I don't think that helps much.  Unless Google is willing to own any
bug reports that come in on CMS when it is turned on.  My main
concern is long term support.   And whether we would have to
do our testing with it both turned on and turned off.  Both those
seem like a big commitment on Oracle's part.  Only my opinion,
of course.

Jon

>
> Jeremy
>
> On Wed, Jun 3, 2015 at 2:32 AM, Volker Simonis 
> <volker.simonis at gmail.com <mailto:volker.simonis at gmail.com>> wrote:
>
>     Hi Jeremy,
>
>     thanks for your answer. Maybe you misunderstood Ivan but it is
>     definitely his and our (i.e. SAP's) intention to make this part of
>     the OpenJDK.
>
>     I can also fully understand the pains you have with integrating
>     changes into the HotSpot. We've gone through this painful process
>     with our ppc64 port a year ago but we think it will finally pay
>     off. After all you (i.e. Google) used it as the base for your
>     ppc64le port (Alexander Smundak contributed most of the
>     LE-specific changes).
>
>     We're facing the exactly same problems regarding the support of
>     proprietary changes in a downstream version of the OpenJDK. That
>     was actually the main reason why we started contributing to the
>     OpenJDK. What we want to kindly ask you is to make your changes
>     and extensions more transparently visible to the community if you
>     intend to integrate them into the OpenJDK. If Hiroshi had posted
>     his changes directly to the hotspot mailing lists instead of
>     sending them privately to Jon I'm pretty sure we would have picked
>     them up. And maybe together we would have managed to push them
>     through (we have some reviewers as well :). After all, there are
>     already quite a lot of features in the OpenJDK which are supported
>     and maintained by the community (i.e. RedHat, IBM, SAP, ...) and
>     not by Oracle.
>
>     I don't have much experience with other Open Source projects, but
>     with OpenJDK and especially HotSpot it's pretty pointless to try
>     to contribute something by writing mails like "..I've implemented
>     this cool feature XXX. If somebody is interested I can post more
>     details..". You'll almost never get any feedback to such kind of
>     requests. You'll at least have to post a complete buildabel and
>     runnable patch against the head revision. I know that's a lot of
>     overhead and we're having this very same discussion if it is worth
>     doing it nearly every day for most of the features/additions we
>     are implementing. Nevertheless, I think it is worth doing it
>     although it sometimes may take years until you get your changes
>     back in the version you are using productively.
>
>     So coming back to Ivan's initial request, we would kindly ask you
>     to just publish your changes in their current form against your
>     current version (8u45 or whatsoever). We'll take a look at them an
>     may invest some time and effort (also jointly with you if desired)
>     to forward port them to 9 and propose them for integration into
>     the head revision if that makes sense. Thus that sound reasonable
>     to you?
>
>     Thank you and best regards,
>     Volker
>
>
>     On Tue, Jun 2, 2015 at 5:00 PM, Jeremy Manson
>     <jeremymanson at google.com <mailto:jeremymanson at google.com>> wrote:
>
>         We'd much rather it be part of OpenJDK, because it is large
>         and fairly complicated, so that would save us time and energy
>         forward porting it every year or so. It's pretty large (for
>         8u45, it is a 2600 line change and touches 20 files), so it
>         tends to be a lengthy and complicated forward port.  We would
>         much rather spend that time doing something more interesting.
>
>         Just posting an updated patch whenever we need to update it
>         would be a little annoying (although not impossible).
>
>         Hiroshi Yamauchi tried to offer it to (I think it was) Jon
>         Masamitsu a few years ago, and Jon didn't know anyone with the
>         bandwidth to review it.  (It was around the same time we
>         contributed the parallel initial mark).  I believe that was
>         when CMS was unowned for a while.  Perhaps there's a different
>         story now?  Oracle volunteers?
>
>         (Procedural: One problem is that Hotspot reviews have to go
>         through someone at Oracle.  We had it written by an OpenJDK
>         author and reviewed by a reviewer.  We've been using it in
>         production for years without too many headaches (although we
>         have yet to inflict^Wlaunch 8u45, which has the latest
>         updates, widely).  I totally understand that Oracle folks are
>         the ones who have to pay the price for a bad check-in, but
>         it's a pretty big hurdle for us.)
>
>         Jeremy
>
>         On Tue, Jun 2, 2015 at 6:12 AM, Galkin, Ivan
>         <ivan.galkin at sap.com <mailto:ivan.galkin at sap.com>> wrote:
>
>             Hi,
>
>             in the thread „JEP 248: Make G1 the Default Garbage
>             Collector” [1] there is an email from Jeremy Manson, who
>             mentions the enhancements made by Google to improve CMS.
>
>             Especially the parallelizing of full compaction is a great
>             improvement and we at SAP see the strong demand of it.
>
>             @Jeremy: Are these changes published somewhere? May we
>             have insight into the diffs you've made? We could
>             collaborate in order to try to bring your changes into
>             OpenJDK.
>
>             Kind regards and thank you in advance,
>
>             Ivan
>
>             [1]
>             http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-June/018740.html
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20150605/d8e7ed8f/attachment.htm>