Parallelizing the full compaction of CMS

Wed Jun 3 20:00:02 UTC 2015

I'm not on hotspot-gc-dev, so I'm not seeing the responses.  But I looked
online...

Jon says that "My understanding was that the change was based on a
parallelization of the serial mark-sweep-compact (used by CMS for full
collections). If it was based on the parallel-old compaction used by
ParallelGC, then my bad.  It should have been reviewed."

His original understanding was correct - we parallelized the serial
mark-sweep-compact used by CMS.  I'm sorry if I misled anyone there.

Otherwise, thanks for your thoughtful reply, Volker.  In my experience of
open source projects, if you can't get a maintainer to agree to consider
your patch, then there is really very little that you can do.  And I
totally get that maintainers don't want / have the time for the burden of
understanding huge amounts of complicated code they didn't write!

So, the question comes down to what "community supported" means in this
case.  We can't actually submit anything to Hotspot without approval from
someone within Oracle.  We have a vested interest in keeping this patch
working, but, for major JDK releases, we really only start stress testing
to identify problems with it when we upgrade to the latest versions of the
JDK, which usually happens some months after Oracle releases it.  That
could cause some issues for upstream.

The upside is that I think we'd be willing to do what it takes, if getting
it upstreamed is a possibility.  This patch is a huge nuisance, and it is
much simpler not to have to do a week or two's worth of forward porting
every year.  And whatever problems arise are problems that we have to fix
anyway.

Jon, if you are particularly worried, note that we can keep it hidden
behind a flag without too much difficulty (and did for a year or two).

Jeremy

On Wed, Jun 3, 2015 at 2:32 AM, Volker Simonis <volker.simonis at gmail.com>
wrote:

> Hi Jeremy,
>
> thanks for your answer. Maybe you misunderstood Ivan but it is definitely
> his and our (i.e. SAP's) intention to make this part of the OpenJDK.
>
> I can also fully understand the pains you have with integrating changes
> into the HotSpot. We've gone through this painful process with our ppc64
> port a year ago but we think it will finally pay off. After all you (i.e.
> Google) used it as the base for your ppc64le port (Alexander Smundak
> contributed most of the LE-specific changes).
>
> We're facing the exactly same problems regarding the support of
> proprietary changes in a downstream version of the OpenJDK. That was
> actually the main reason why we started contributing to the OpenJDK. What
> we want to kindly ask you is to make your changes and extensions more
> transparently visible to the community if you intend to integrate them into
> the OpenJDK. If Hiroshi had posted his changes directly to the hotspot
> mailing lists instead of sending them privately to Jon I'm pretty sure we
> would have picked them up. And maybe together we would have managed to push
> them through (we have some reviewers as well :). After all, there are
> already quite a lot of features in the OpenJDK which are supported and
> maintained by the community (i.e. RedHat, IBM, SAP, ...) and not by Oracle.
>
> I don't have much experience with other Open Source projects, but with
> OpenJDK and especially HotSpot it's pretty pointless to try to contribute
> something by writing mails like "..I've implemented this cool feature XXX.
> If somebody is interested I can post more details..". You'll almost never
> get any feedback to such kind of requests. You'll at least have to post a
> complete buildabel and runnable patch against the head revision. I know
> that's a lot of overhead and we're having this very same discussion if it
> is worth doing it nearly every day for most of the features/additions we
> are implementing. Nevertheless, I think it is worth doing it although it
> sometimes may take years until you get your changes back in the version you
> are using productively.
>
> So coming back to Ivan's initial request, we would kindly ask you to just
> publish your changes in their current form against your current version
> (8u45 or whatsoever). We'll take a look at them an may invest some time and
> effort (also jointly with you if desired) to forward port them to 9 and
> propose them for integration into the head revision if that makes sense.
> Thus that sound reasonable to you?
>
> Thank you and best regards,
> Volker
>
>
> On Tue, Jun 2, 2015 at 5:00 PM, Jeremy Manson <jeremymanson at google.com>
> wrote:
>
>> We'd much rather it be part of OpenJDK, because it is large and fairly
>> complicated, so that would save us time and energy forward porting it every
>> year or so.  It's pretty large (for 8u45, it is a 2600 line change and
>> touches 20 files), so it tends to be a lengthy and complicated forward
>> port.  We would much rather spend that time doing something more
>> interesting.
>>
>> Just posting an updated patch whenever we need to update it would be a
>> little annoying (although not impossible).
>>
>> Hiroshi Yamauchi tried to offer it to (I think it was) Jon Masamitsu a
>> few years ago, and Jon didn't know anyone with the bandwidth to review it.
>>  (It was around the same time we contributed the parallel initial mark).  I
>> believe that was when CMS was unowned for a while.  Perhaps there's a
>> different story now?  Oracle volunteers?
>>
>> (Procedural: One problem is that Hotspot reviews have to go through
>> someone at Oracle.  We had it written by an OpenJDK author and reviewed by
>> a reviewer.  We've been using it in production for years without too many
>> headaches (although we have yet to inflict^Wlaunch 8u45, which has the
>> latest updates, widely).  I totally understand that Oracle folks are the
>> ones who have to pay the price for a bad check-in, but it's a pretty big
>> hurdle for us.)
>>
>> Jeremy
>>
>> On Tue, Jun 2, 2015 at 6:12 AM, Galkin, Ivan <ivan.galkin at sap.com> wrote:
>>
>>>  Hi,
>>>
>>>
>>>
>>> in the thread „JEP 248: Make G1 the Default Garbage Collector” [1] there
>>> is an email from Jeremy Manson, who mentions the enhancements made by
>>> Google to improve CMS.
>>>
>>> Especially the parallelizing of full compaction is a great improvement
>>> and we at SAP see the strong demand of it.
>>>
>>>
>>>
>>> @Jeremy: Are these changes published somewhere? May we have insight into
>>> the diffs you've made? We could collaborate in order to try to bring your
>>> changes into OpenJDK.
>>>
>>>
>>>
>>> Kind regards and thank you in advance,
>>>
>>> Ivan
>>>
>>>
>>>
>>> [1]
>>> http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-June/018740.html
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20150603/85ea3d6c/attachment.htm>