Parallelizing the full compaction of CMS

Jon Masamitsu jon.masamitsu at oracle.com
Mon Jun 8 16:00:19 UTC 2015



On 6/8/2015 4:11 AM, Jungwoo Ha wrote:
> Hi Volker,
>
> I am going to post the code some time this week.
> Stay tuned.

Jungwoo,

Will this be based on jdk8 or jdk9?

Jon

>
> --Jungwoo
>
>
> On Mon, Jun 8, 2015 at 3:04 AM, Volker Simonis 
> <volker.simonis at gmail.com <mailto:volker.simonis at gmail.com>> wrote:
>
>     Hi John, Jeremy,
>
>     @John: sorry, but I don't see what's the big problem behind "supported
>     and maintained by the community". We already have quite some code in
>     the HotSpot which is developed and maintained that way. Take for
>     example the C++-Interpreter. It has not been used and supported by
>     Sun/Oracle since 1.4 where it was used for the Itanium port.
>     Nevertheless it is quite useful for HotSpot ports to new platforms. We
>     at SAP used it for our intitial PowerPC port for example. And of
>     course we supported it. This includes both fixes as well as the
>     implementation of new features (e.g. supports for JSR 292). Another
>     user of the C++-interpreter is the Zero-port which is supported mainly
>     by RedHat. Again, Oracle does neither build nor test the Zero-port.
>     This is all done by RedHat. But it is still a crucial part of the
>     OpenJDK for platforms like Linux on zArch, MIPS, Itanium,....
>
>     Then we (i.e. SAP and IBM) have our ports (Linux/PPC64, AIX) and
>     RedHat have their AArch64 port. Neither of them is supported by Oracle
>     but all of them are pretty well supported by their maintainers. We
>     have our own platform  categories in the bug system, our OpenJDK
>     mailing lists. We do nightly builds and tests and we take all this
>     pretty seriously. And that all imposes nearly zero overhead on Oracle
>     but rather helps to strengthen the VM on all platforms (e.g. we
>     contributed quite some GC-fixes which only showed up on a weak memory
>     architecture like Power or helped to clean up the code because we use
>     different compiler on other platforms).
>
>     I don't think it is appropriate for an Open Source project to reject a
>     new feature just because one of the maintainers doesn't want it. And
>     the OpenJDK is an OpenSource project with a distinct development model
>     as specified in the OpenJDK bylaws (http://openjdk.java.net/bylaws).
>     But I don't want to go on collision course. Until now the cooperation
>     between the various contributors in the OpenJDK has always worked
>     quite well. And if a feature can really be hidden behind a flag
>     without impacting the other configurations that's great. Of course
>     Oracle would not have to maintain or even test it (there already exist
>     a myriad of unsupported configurations today :). And I think it is
>     self-evident that contributors feel responsible for bugs in their code
>     and fix them (as we already do today). So the only commitment on
>     Oracle part would be to help with the initial review process which I'd
>     consider acceptable.
>
>      @Jeremy: could you please post some code so we can see what we're
>     really talking about here. It seems that until now nobody ever saw
>     these changes. And maybe it helps if several parties are trying
>     jointly to 'convince' the maintainer to consider a patch :)
>
>     Regards,
>     Volker
>
>
>     On Fri, Jun 5, 2015 at 5:56 PM, Jon Masamitsu
>     <jon.masamitsu at oracle.com <mailto:jon.masamitsu at oracle.com>> wrote:
>     >
>     >
>     > On 6/3/2015 1:00 PM, Jeremy Manson wrote:
>     >
>     > I'm not on hotspot-gc-dev, so I'm not seeing the responses.  But
>     I looked
>     > online...
>     >
>     > Jon says that "My understanding was that the change was based on a
>     > parallelization of the serial mark-sweep-compact (used by CMS
>     for full
>     > collections). If it was based on the parallel-old compaction used by
>     > ParallelGC, then my bad.  It should have been reviewed."
>     >
>     > His original understanding was correct - we parallelized the serial
>     > mark-sweep-compact used by CMS.  I'm sorry if I misled anyone there.
>     >
>     >
>     > No problem.  It gave me a chance to say again that
>     > I'd rather not have to support two implementations of
>     > a parallel full stop-the-world collector.  We have that
>     > issue with the two implementations of a parallel young
>     > collector and I'd really rather not go there again.
>     >
>     >
>     > Otherwise, thanks for your thoughtful reply, Volker.  In my
>     experience of
>     > open source projects, if you can't get a maintainer to agree to
>     consider
>     > your patch, then there is really very little that you can do. 
>     And I totally
>     > get that maintainers don't want / have the time for the burden of
>     > understanding huge amounts of complicated code they didn't write!
>     >
>     > So, the question comes down to what "community supported" means
>     in this
>     > case.  We can't actually submit anything to Hotspot without
>     approval from
>     > someone within Oracle.  We have a vested interest in keeping
>     this patch
>     > working, but, for major JDK releases, we really only start
>     stress testing to
>     > identify problems with it when we upgrade to the latest versions
>     of the JDK,
>     > which usually happens some months after Oracle releases it. 
>     That could
>     > cause some issues for upstream.
>     >
>     > The upside is that I think we'd be willing to do what it takes,
>     if getting
>     > it upstreamed is a possibility.  This patch is a huge nuisance,
>     and it is
>     > much simpler not to have to do a week or two's worth of forward
>     porting
>     > every year.  And whatever problems arise are problems that we
>     have to fix
>     > anyway.
>     >
>     > Jon, if you are particularly worried, note that we can keep it
>     hidden behind
>     > a flag without too much difficulty (and did for a year or two).
>     >
>     >
>     > I don't think that helps much.  Unless Google is willing to own any
>     > bug reports that come in on CMS when it is turned on.  My main
>     > concern is long term support.   And whether we would have to
>     > do our testing with it both turned on and turned off.  Both those
>     > seem like a big commitment on Oracle's part. Only my opinion,
>     > of course.
>     >
>     > Jon
>     >
>     >
>     > Jeremy
>     >
>     > On Wed, Jun 3, 2015 at 2:32 AM, Volker Simonis
>     <volker.simonis at gmail.com <mailto:volker.simonis at gmail.com>>
>     > wrote:
>     >>
>     >> Hi Jeremy,
>     >>
>     >> thanks for your answer. Maybe you misunderstood Ivan but it is
>     definitely
>     >> his and our (i.e. SAP's) intention to make this part of the
>     OpenJDK.
>     >>
>     >> I can also fully understand the pains you have with integrating
>     changes
>     >> into the HotSpot. We've gone through this painful process with
>     our ppc64
>     >> port a year ago but we think it will finally pay off. After all
>     you (i.e.
>     >> Google) used it as the base for your ppc64le port (Alexander
>     Smundak
>     >> contributed most of the LE-specific changes).
>     >>
>     >> We're facing the exactly same problems regarding the support of
>     >> proprietary changes in a downstream version of the OpenJDK.
>     That was
>     >> actually the main reason why we started contributing to the
>     OpenJDK. What we
>     >> want to kindly ask you is to make your changes and extensions more
>     >> transparently visible to the community if you intend to
>     integrate them into
>     >> the OpenJDK. If Hiroshi had posted his changes directly to the
>     hotspot
>     >> mailing lists instead of sending them privately to Jon I'm
>     pretty sure we
>     >> would have picked them up. And maybe together we would have
>     managed to push
>     >> them through (we have some reviewers as well :). After all,
>     there are
>     >> already quite a lot of features in the OpenJDK which are
>     supported and
>     >> maintained by the community (i.e. RedHat, IBM, SAP, ...) and
>     not by Oracle.
>     >>
>     >> I don't have much experience with other Open Source projects,
>     but with
>     >> OpenJDK and especially HotSpot it's pretty pointless to try to
>     contribute
>     >> something by writing mails like "..I've implemented this cool
>     feature XXX.
>     >> If somebody is interested I can post more details..". You'll
>     almost never
>     >> get any feedback to such kind of requests. You'll at least have
>     to post a
>     >> complete buildabel and runnable patch against the head
>     revision. I know
>     >> that's a lot of overhead and we're having this very same
>     discussion if it is
>     >> worth doing it nearly every day for most of the
>     features/additions we are
>     >> implementing. Nevertheless, I think it is worth doing it
>     although it
>     >> sometimes may take years until you get your changes back in the
>     version you
>     >> are using productively.
>     >>
>     >> So coming back to Ivan's initial request, we would kindly ask
>     you to just
>     >> publish your changes in their current form against your current
>     version
>     >> (8u45 or whatsoever). We'll take a look at them an may invest
>     some time and
>     >> effort (also jointly with you if desired) to forward port them
>     to 9 and
>     >> propose them for integration into the head revision if that
>     makes sense.
>     >> Thus that sound reasonable to you?
>     >>
>     >> Thank you and best regards,
>     >> Volker
>     >>
>     >>
>     >> On Tue, Jun 2, 2015 at 5:00 PM, Jeremy Manson
>     <jeremymanson at google.com <mailto:jeremymanson at google.com>>
>     >> wrote:
>     >>>
>     >>> We'd much rather it be part of OpenJDK, because it is large
>     and fairly
>     >>> complicated, so that would save us time and energy forward
>     porting it every
>     >>> year or so.  It's pretty large (for 8u45, it is a 2600 line
>     change and
>     >>> touches 20 files), so it tends to be a lengthy and complicated
>     forward port.
>     >>> We would much rather spend that time doing something more
>     interesting.
>     >>>
>     >>> Just posting an updated patch whenever we need to update it
>     would be a
>     >>> little annoying (although not impossible).
>     >>>
>     >>> Hiroshi Yamauchi tried to offer it to (I think it was) Jon
>     Masamitsu a
>     >>> few years ago, and Jon didn't know anyone with the bandwidth
>     to review it.
>     >>> (It was around the same time we contributed the parallel
>     initial mark).  I
>     >>> believe that was when CMS was unowned for a while.  Perhaps
>     there's a
>     >>> different story now?  Oracle volunteers?
>     >>>
>     >>> (Procedural: One problem is that Hotspot reviews have to go
>     through
>     >>> someone at Oracle.  We had it written by an OpenJDK author and
>     reviewed by a
>     >>> reviewer.  We've been using it in production for years without
>     too many
>     >>> headaches (although we have yet to inflict^Wlaunch 8u45, which
>     has the
>     >>> latest updates, widely).  I totally understand that Oracle
>     folks are the
>     >>> ones who have to pay the price for a bad check-in, but it's a
>     pretty big
>     >>> hurdle for us.)
>     >>>
>     >>> Jeremy
>     >>>
>     >>> On Tue, Jun 2, 2015 at 6:12 AM, Galkin, Ivan
>     <ivan.galkin at sap.com <mailto:ivan.galkin at sap.com>> wrote:
>     >>>>
>     >>>> Hi,
>     >>>>
>     >>>>
>     >>>>
>     >>>> in the thread „JEP 248: Make G1 the Default Garbage
>     Collector” [1] there
>     >>>> is an email from Jeremy Manson, who mentions the enhancements
>     made by Google
>     >>>> to improve CMS.
>     >>>>
>     >>>> Especially the parallelizing of full compaction is a great
>     improvement
>     >>>> and we at SAP see the strong demand of it.
>     >>>>
>     >>>>
>     >>>>
>     >>>> @Jeremy: Are these changes published somewhere? May we have
>     insight into
>     >>>> the diffs you've made? We could collaborate in order to try
>     to bring your
>     >>>> changes into OpenJDK.
>     >>>>
>     >>>>
>     >>>>
>     >>>> Kind regards and thank you in advance,
>     >>>>
>     >>>> Ivan
>     >>>>
>     >>>>
>     >>>>
>     >>>> [1]
>     >>>>
>     http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-June/018740.html
>     >>>
>     >>>
>     >>
>     >
>     >
>
>
>
>
> -- 
> Jungwoo Ha | Java Platform Team | jwha at google.com <mailto:jwha at google.com>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20150608/de64a1a4/attachment.htm>


More information about the hotspot-gc-dev mailing list