Parallelizing the full compaction of CMS

[Volker Simonis]

Hi John,

that's exactly what I wanted to say.

Thanks,
Volker


On Mon, Jun 8, 2015 at 5:58 PM, Jon Masamitsu <jon.masamitsu at oracle.com> wrote:
>
>
> On 6/8/2015 3:04 AM, Volker Simonis wrote:
>>
>> Hi John, Jeremy,
>>
>> @John: sorry, but I don't see what's the big problem behind "supported
>> and maintained by the community". We already have quite some code in
>> the HotSpot which is developed and maintained that way. Take for
>> example the C++-Interpreter. It has not been used and supported by
>> Sun/Oracle since 1.4 where it was used for the Itanium port.
>> Nevertheless it is quite useful for HotSpot ports to new platforms. We
>> at SAP used it for our intitial PowerPC port for example. And of
>> course we supported it. This includes both fixes as well as the
>> implementation of new features (e.g. supports for JSR 292). Another
>> user of the C++-interpreter is the Zero-port which is supported mainly
>> by RedHat. Again, Oracle does neither build nor test the Zero-port.
>> This is all done by RedHat. But it is still a crucial part of the
>> OpenJDK for platforms like Linux on zArch, MIPS, Itanium,....
>>
>> Then we (i.e. SAP and IBM) have our ports (Linux/PPC64, AIX) and
>> RedHat have their AArch64 port. Neither of them is supported by Oracle
>> but all of them are pretty well supported by their maintainers. We
>> have our own platform  categories in the bug system, our OpenJDK
>> mailing lists. We do nightly builds and tests and we take all this
>> pretty seriously. And that all imposes nearly zero overhead on Oracle
>> but rather helps to strengthen the VM on all platforms (e.g. we
>> contributed quite some GC-fixes which only showed up on a weak memory
>> architecture like Power or helped to clean up the code because we use
>> different compiler on other platforms).
>>
>> I don't think it is appropriate for an Open Source project to reject a
>> new feature just because one of the maintainers doesn't want it. And
>> the OpenJDK is an OpenSource project with a distinct development model
>> as specified in the OpenJDK bylaws (http://openjdk.java.net/bylaws).
>> But I don't want to go on collision course. Until now the cooperation
>> between the various contributors in the OpenJDK has always worked
>> quite well. And if a feature can really be hidden behind a flag
>> without impacting the other configurations that's great. Of course
>> Oracle would not have to maintain or even test it (there already exist
>> a myriad of unsupported configurations today :). And I think it is
>> self-evident that contributors feel responsible for bugs in their code
>> and fix them (as we already do today). So the only commitment on
>> Oracle part would be to help with the initial review process which I'd
>> consider acceptable.
>
>
> Volker,
>
> What I think you're saying is that there have been contributions
> that are not supported by Oracle and have no cost for Oracle.
> And that this parallel full compaction for CMS would be another.
>
> If a bug is reported where parallel full compaction with CMS is turned on
> and Oracle is not expected to look at the bug, then I have no problem with
> the integration of such a change after the usual code review.   And as you
> say, there is no need for Oracle to test with it turned on.   As you note
> I'm only one maintainer.   Oracle may have other opinions.
>
> Jon
>
>
>>
>>   @Jeremy: could you please post some code so we can see what we're
>> really talking about here. It seems that until now nobody ever saw
>> these changes. And maybe it helps if several parties are trying
>> jointly to 'convince' the maintainer to consider a patch :)
>>
>> Regards,
>> Volker
>>
>>
>> On Fri, Jun 5, 2015 at 5:56 PM, Jon Masamitsu <jon.masamitsu at oracle.com>
>> wrote:
>>>
>>>
>>> On 6/3/2015 1:00 PM, Jeremy Manson wrote:
>>>
>>> I'm not on hotspot-gc-dev, so I'm not seeing the responses.  But I looked
>>> online...
>>>
>>> Jon says that "My understanding was that the change was based on a
>>> parallelization of the serial mark-sweep-compact (used by CMS for full
>>> collections). If it was based on the parallel-old compaction used by
>>> ParallelGC, then my bad.  It should have been reviewed."
>>>
>>> His original understanding was correct - we parallelized the serial
>>> mark-sweep-compact used by CMS.  I'm sorry if I misled anyone there.
>>>
>>>
>>> No problem.  It gave me a chance to say again that
>>> I'd rather not have to support two implementations of
>>> a parallel full stop-the-world collector.  We have that
>>> issue with the two implementations of a parallel young
>>> collector and I'd really rather not go there again.
>>>
>>>
>>> Otherwise, thanks for your thoughtful reply, Volker.  In my experience of
>>> open source projects, if you can't get a maintainer to agree to consider
>>> your patch, then there is really very little that you can do.  And I
>>> totally
>>> get that maintainers don't want / have the time for the burden of
>>> understanding huge amounts of complicated code they didn't write!
>>>
>>> So, the question comes down to what "community supported" means in this
>>> case.  We can't actually submit anything to Hotspot without approval from
>>> someone within Oracle.  We have a vested interest in keeping this patch
>>> working, but, for major JDK releases, we really only start stress testing
>>> to
>>> identify problems with it when we upgrade to the latest versions of the
>>> JDK,
>>> which usually happens some months after Oracle releases it.  That could
>>> cause some issues for upstream.
>>>
>>> The upside is that I think we'd be willing to do what it takes, if
>>> getting
>>> it upstreamed is a possibility.  This patch is a huge nuisance, and it is
>>> much simpler not to have to do a week or two's worth of forward porting
>>> every year.  And whatever problems arise are problems that we have to fix
>>> anyway.
>>>
>>> Jon, if you are particularly worried, note that we can keep it hidden
>>> behind
>>> a flag without too much difficulty (and did for a year or two).
>>>
>>>
>>> I don't think that helps much.  Unless Google is willing to own any
>>> bug reports that come in on CMS when it is turned on.  My main
>>> concern is long term support.   And whether we would have to
>>> do our testing with it both turned on and turned off.  Both those
>>> seem like a big commitment on Oracle's part.  Only my opinion,
>>> of course.
>>>
>>> Jon
>>>
>>>
>>> Jeremy
>>>
>>> On Wed, Jun 3, 2015 at 2:32 AM, Volker Simonis <volker.simonis at gmail.com>
>>> wrote:
>>>>
>>>> Hi Jeremy,
>>>>
>>>> thanks for your answer. Maybe you misunderstood Ivan but it is
>>>> definitely
>>>> his and our (i.e. SAP's) intention to make this part of the OpenJDK.
>>>>
>>>> I can also fully understand the pains you have with integrating changes
>>>> into the HotSpot. We've gone through this painful process with our ppc64
>>>> port a year ago but we think it will finally pay off. After all you
>>>> (i.e.
>>>> Google) used it as the base for your ppc64le port (Alexander Smundak
>>>> contributed most of the LE-specific changes).
>>>>
>>>> We're facing the exactly same problems regarding the support of
>>>> proprietary changes in a downstream version of the OpenJDK. That was
>>>> actually the main reason why we started contributing to the OpenJDK.
>>>> What we
>>>> want to kindly ask you is to make your changes and extensions more
>>>> transparently visible to the community if you intend to integrate them
>>>> into
>>>> the OpenJDK. If Hiroshi had posted his changes directly to the hotspot
>>>> mailing lists instead of sending them privately to Jon I'm pretty sure
>>>> we
>>>> would have picked them up. And maybe together we would have managed to
>>>> push
>>>> them through (we have some reviewers as well :). After all, there are
>>>> already quite a lot of features in the OpenJDK which are supported and
>>>> maintained by the community (i.e. RedHat, IBM, SAP, ...) and not by
>>>> Oracle.
>>>>
>>>> I don't have much experience with other Open Source projects, but with
>>>> OpenJDK and especially HotSpot it's pretty pointless to try to
>>>> contribute
>>>> something by writing mails like "..I've implemented this cool feature
>>>> XXX.
>>>> If somebody is interested I can post more details..". You'll almost
>>>> never
>>>> get any feedback to such kind of requests. You'll at least have to post
>>>> a
>>>> complete buildabel and runnable patch against the head revision. I know
>>>> that's a lot of overhead and we're having this very same discussion if
>>>> it is
>>>> worth doing it nearly every day for most of the features/additions we
>>>> are
>>>> implementing. Nevertheless, I think it is worth doing it although it
>>>> sometimes may take years until you get your changes back in the version
>>>> you
>>>> are using productively.
>>>>
>>>> So coming back to Ivan's initial request, we would kindly ask you to
>>>> just
>>>> publish your changes in their current form against your current version
>>>> (8u45 or whatsoever). We'll take a look at them an may invest some time
>>>> and
>>>> effort (also jointly with you if desired) to forward port them to 9 and
>>>> propose them for integration into the head revision if that makes sense.
>>>> Thus that sound reasonable to you?
>>>>
>>>> Thank you and best regards,
>>>> Volker
>>>>
>>>>
>>>> On Tue, Jun 2, 2015 at 5:00 PM, Jeremy Manson <jeremymanson at google.com>
>>>> wrote:
>>>>>
>>>>> We'd much rather it be part of OpenJDK, because it is large and fairly
>>>>> complicated, so that would save us time and energy forward porting it
>>>>> every
>>>>> year or so.  It's pretty large (for 8u45, it is a 2600 line change and
>>>>> touches 20 files), so it tends to be a lengthy and complicated forward
>>>>> port.
>>>>> We would much rather spend that time doing something more interesting.
>>>>>
>>>>> Just posting an updated patch whenever we need to update it would be a
>>>>> little annoying (although not impossible).
>>>>>
>>>>> Hiroshi Yamauchi tried to offer it to (I think it was) Jon Masamitsu a
>>>>> few years ago, and Jon didn't know anyone with the bandwidth to review
>>>>> it.
>>>>> (It was around the same time we contributed the parallel initial mark).
>>>>> I
>>>>> believe that was when CMS was unowned for a while.  Perhaps there's a
>>>>> different story now?  Oracle volunteers?
>>>>>
>>>>> (Procedural: One problem is that Hotspot reviews have to go through
>>>>> someone at Oracle.  We had it written by an OpenJDK author and reviewed
>>>>> by a
>>>>> reviewer.  We've been using it in production for years without too many
>>>>> headaches (although we have yet to inflict^Wlaunch 8u45, which has the
>>>>> latest updates, widely).  I totally understand that Oracle folks are
>>>>> the
>>>>> ones who have to pay the price for a bad check-in, but it's a pretty
>>>>> big
>>>>> hurdle for us.)
>>>>>
>>>>> Jeremy
>>>>>
>>>>> On Tue, Jun 2, 2015 at 6:12 AM, Galkin, Ivan <ivan.galkin at sap.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> in the thread „JEP 248: Make G1 the Default Garbage Collector” [1]
>>>>>> there
>>>>>> is an email from Jeremy Manson, who mentions the enhancements made by
>>>>>> Google
>>>>>> to improve CMS.
>>>>>>
>>>>>> Especially the parallelizing of full compaction is a great improvement
>>>>>> and we at SAP see the strong demand of it.
>>>>>>
>>>>>>
>>>>>>
>>>>>> @Jeremy: Are these changes published somewhere? May we have insight
>>>>>> into
>>>>>> the diffs you've made? We could collaborate in order to try to bring
>>>>>> your
>>>>>> changes into OpenJDK.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Kind regards and thank you in advance,
>>>>>>
>>>>>> Ivan
>>>>>>
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>> http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-June/018740.html
>>>>>
>>>>>
>>>
>