Parallelizing the full compaction of CMS

[Jungwoo Ha]

Hi Volker,

I am going to post the code some time this week.
Stay tuned.

--Jungwoo


On Mon, Jun 8, 2015 at 3:04 AM, Volker Simonis <volker.simonis at gmail.com>
wrote:

> Hi John, Jeremy,
>
> @John: sorry, but I don't see what's the big problem behind "supported
> and maintained by the community". We already have quite some code in
> the HotSpot which is developed and maintained that way. Take for
> example the C++-Interpreter. It has not been used and supported by
> Sun/Oracle since 1.4 where it was used for the Itanium port.
> Nevertheless it is quite useful for HotSpot ports to new platforms. We
> at SAP used it for our intitial PowerPC port for example. And of
> course we supported it. This includes both fixes as well as the
> implementation of new features (e.g. supports for JSR 292). Another
> user of the C++-interpreter is the Zero-port which is supported mainly
> by RedHat. Again, Oracle does neither build nor test the Zero-port.
> This is all done by RedHat. But it is still a crucial part of the
> OpenJDK for platforms like Linux on zArch, MIPS, Itanium,....
>
> Then we (i.e. SAP and IBM) have our ports (Linux/PPC64, AIX) and
> RedHat have their AArch64 port. Neither of them is supported by Oracle
> but all of them are pretty well supported by their maintainers. We
> have our own platform  categories in the bug system, our OpenJDK
> mailing lists. We do nightly builds and tests and we take all this
> pretty seriously. And that all imposes nearly zero overhead on Oracle
> but rather helps to strengthen the VM on all platforms (e.g. we
> contributed quite some GC-fixes which only showed up on a weak memory
> architecture like Power or helped to clean up the code because we use
> different compiler on other platforms).
>
> I don't think it is appropriate for an Open Source project to reject a
> new feature just because one of the maintainers doesn't want it. And
> the OpenJDK is an OpenSource project with a distinct development model
> as specified in the OpenJDK bylaws (http://openjdk.java.net/bylaws).
> But I don't want to go on collision course. Until now the cooperation
> between the various contributors in the OpenJDK has always worked
> quite well. And if a feature can really be hidden behind a flag
> without impacting the other configurations that's great. Of course
> Oracle would not have to maintain or even test it (there already exist
> a myriad of unsupported configurations today :). And I think it is
> self-evident that contributors feel responsible for bugs in their code
> and fix them (as we already do today). So the only commitment on
> Oracle part would be to help with the initial review process which I'd
> consider acceptable.
>
>  @Jeremy: could you please post some code so we can see what we're
> really talking about here. It seems that until now nobody ever saw
> these changes. And maybe it helps if several parties are trying
> jointly to 'convince' the maintainer to consider a patch :)
>
> Regards,
> Volker
>
>
> On Fri, Jun 5, 2015 at 5:56 PM, Jon Masamitsu <jon.masamitsu at oracle.com>
> wrote:
> >
> >
> > On 6/3/2015 1:00 PM, Jeremy Manson wrote:
> >
> > I'm not on hotspot-gc-dev, so I'm not seeing the responses.  But I looked
> > online...
> >
> > Jon says that "My understanding was that the change was based on a
> > parallelization of the serial mark-sweep-compact (used by CMS for full
> > collections). If it was based on the parallel-old compaction used by
> > ParallelGC, then my bad.  It should have been reviewed."
> >
> > His original understanding was correct - we parallelized the serial
> > mark-sweep-compact used by CMS.  I'm sorry if I misled anyone there.
> >
> >
> > No problem.  It gave me a chance to say again that
> > I'd rather not have to support two implementations of
> > a parallel full stop-the-world collector.  We have that
> > issue with the two implementations of a parallel young
> > collector and I'd really rather not go there again.
> >
> >
> > Otherwise, thanks for your thoughtful reply, Volker.  In my experience of
> > open source projects, if you can't get a maintainer to agree to consider
> > your patch, then there is really very little that you can do.  And I
> totally
> > get that maintainers don't want / have the time for the burden of
> > understanding huge amounts of complicated code they didn't write!
> >
> > So, the question comes down to what "community supported" means in this
> > case.  We can't actually submit anything to Hotspot without approval from
> > someone within Oracle.  We have a vested interest in keeping this patch
> > working, but, for major JDK releases, we really only start stress
> testing to
> > identify problems with it when we upgrade to the latest versions of the
> JDK,
> > which usually happens some months after Oracle releases it.  That could
> > cause some issues for upstream.
> >
> > The upside is that I think we'd be willing to do what it takes, if
> getting
> > it upstreamed is a possibility.  This patch is a huge nuisance, and it is
> > much simpler not to have to do a week or two's worth of forward porting
> > every year.  And whatever problems arise are problems that we have to fix
> > anyway.
> >
> > Jon, if you are particularly worried, note that we can keep it hidden
> behind
> > a flag without too much difficulty (and did for a year or two).
> >
> >
> > I don't think that helps much.  Unless Google is willing to own any
> > bug reports that come in on CMS when it is turned on.  My main
> > concern is long term support.   And whether we would have to
> > do our testing with it both turned on and turned off.  Both those
> > seem like a big commitment on Oracle's part.  Only my opinion,
> > of course.
> >
> > Jon
> >
> >
> > Jeremy
> >
> > On Wed, Jun 3, 2015 at 2:32 AM, Volker Simonis <volker.simonis at gmail.com
> >
> > wrote:
> >>
> >> Hi Jeremy,
> >>
> >> thanks for your answer. Maybe you misunderstood Ivan but it is
> definitely
> >> his and our (i.e. SAP's) intention to make this part of the OpenJDK.
> >>
> >> I can also fully understand the pains you have with integrating changes
> >> into the HotSpot. We've gone through this painful process with our ppc64
> >> port a year ago but we think it will finally pay off. After all you
> (i.e.
> >> Google) used it as the base for your ppc64le port (Alexander Smundak
> >> contributed most of the LE-specific changes).
> >>
> >> We're facing the exactly same problems regarding the support of
> >> proprietary changes in a downstream version of the OpenJDK. That was
> >> actually the main reason why we started contributing to the OpenJDK.
> What we
> >> want to kindly ask you is to make your changes and extensions more
> >> transparently visible to the community if you intend to integrate them
> into
> >> the OpenJDK. If Hiroshi had posted his changes directly to the hotspot
> >> mailing lists instead of sending them privately to Jon I'm pretty sure
> we
> >> would have picked them up. And maybe together we would have managed to
> push
> >> them through (we have some reviewers as well :). After all, there are
> >> already quite a lot of features in the OpenJDK which are supported and
> >> maintained by the community (i.e. RedHat, IBM, SAP, ...) and not by
> Oracle.
> >>
> >> I don't have much experience with other Open Source projects, but with
> >> OpenJDK and especially HotSpot it's pretty pointless to try to
> contribute
> >> something by writing mails like "..I've implemented this cool feature
> XXX.
> >> If somebody is interested I can post more details..". You'll almost
> never
> >> get any feedback to such kind of requests. You'll at least have to post
> a
> >> complete buildabel and runnable patch against the head revision. I know
> >> that's a lot of overhead and we're having this very same discussion if
> it is
> >> worth doing it nearly every day for most of the features/additions we
> are
> >> implementing. Nevertheless, I think it is worth doing it although it
> >> sometimes may take years until you get your changes back in the version
> you
> >> are using productively.
> >>
> >> So coming back to Ivan's initial request, we would kindly ask you to
> just
> >> publish your changes in their current form against your current version
> >> (8u45 or whatsoever). We'll take a look at them an may invest some time
> and
> >> effort (also jointly with you if desired) to forward port them to 9 and
> >> propose them for integration into the head revision if that makes sense.
> >> Thus that sound reasonable to you?
> >>
> >> Thank you and best regards,
> >> Volker
> >>
> >>
> >> On Tue, Jun 2, 2015 at 5:00 PM, Jeremy Manson <jeremymanson at google.com>
> >> wrote:
> >>>
> >>> We'd much rather it be part of OpenJDK, because it is large and fairly
> >>> complicated, so that would save us time and energy forward porting it
> every
> >>> year or so.  It's pretty large (for 8u45, it is a 2600 line change and
> >>> touches 20 files), so it tends to be a lengthy and complicated forward
> port.
> >>> We would much rather spend that time doing something more interesting.
> >>>
> >>> Just posting an updated patch whenever we need to update it would be a
> >>> little annoying (although not impossible).
> >>>
> >>> Hiroshi Yamauchi tried to offer it to (I think it was) Jon Masamitsu a
> >>> few years ago, and Jon didn't know anyone with the bandwidth to review
> it.
> >>> (It was around the same time we contributed the parallel initial
> mark).  I
> >>> believe that was when CMS was unowned for a while.  Perhaps there's a
> >>> different story now?  Oracle volunteers?
> >>>
> >>> (Procedural: One problem is that Hotspot reviews have to go through
> >>> someone at Oracle.  We had it written by an OpenJDK author and
> reviewed by a
> >>> reviewer.  We've been using it in production for years without too many
> >>> headaches (although we have yet to inflict^Wlaunch 8u45, which has the
> >>> latest updates, widely).  I totally understand that Oracle folks are
> the
> >>> ones who have to pay the price for a bad check-in, but it's a pretty
> big
> >>> hurdle for us.)
> >>>
> >>> Jeremy
> >>>
> >>> On Tue, Jun 2, 2015 at 6:12 AM, Galkin, Ivan <ivan.galkin at sap.com>
> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>>
> >>>>
> >>>> in the thread „JEP 248: Make G1 the Default Garbage Collector” [1]
> there
> >>>> is an email from Jeremy Manson, who mentions the enhancements made by
> Google
> >>>> to improve CMS.
> >>>>
> >>>> Especially the parallelizing of full compaction is a great improvement
> >>>> and we at SAP see the strong demand of it.
> >>>>
> >>>>
> >>>>
> >>>> @Jeremy: Are these changes published somewhere? May we have insight
> into
> >>>> the diffs you've made? We could collaborate in order to try to bring
> your
> >>>> changes into OpenJDK.
> >>>>
> >>>>
> >>>>
> >>>> Kind regards and thank you in advance,
> >>>>
> >>>> Ivan
> >>>>
> >>>>
> >>>>
> >>>> [1]
> >>>>
> http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-June/018740.html
> >>>
> >>>
> >>
> >
> >
>



-- 
Jungwoo Ha | Java Platform Team | jwha at google.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20150608/bfe1affe/attachment.htm>