CMS Garbage collection eating up processor power

[T.K]

I've attached an old gclog where it has similar behavior on the heap.  But
the problem with that is it takes quite a long time, like 20 - 70 seconds.
The server basically went down with customers complaining during that time,
so we had to try CMS, but that doesn't work out well either.

We are having Sun Portal Server 6.2 on those web servers.  Do you know is it
a normal behavior for Portal Server with about 400-500 users per instance?
Thanks,
TK

On Thu, Mar 27, 2008 at 5:10 PM, Jon Masamitsu <Jon.Masamitsu at sun.com>
wrote:

> Late in this log I see
>
> 162417.510: [GC 162417.510: [ParNew: 370176K->11520K(381696K), 0.4232456
> secs] 3055096K->2715938K(3134208K), 0.4240171 secs]
>
> At that point the heap is about 85% full (2715938K/3134208K).  The
> tenured generation is almost completely full.
>
> Do you have similar logs when using the default GC?  We could use them
> to verify the amount of live data.
>
>
> T.K wrote:
> >
> > Hi Jon,
> >
> > Here's the attached gclog.  It starts out fine when we put in the change
> > in the evening.  Eventually, the next morning (around 60000 seconds on
> > gc time) when the load starting to get in, CMS starts to run
> > consectively.  The first CMS fails occur when we bump up the users to
> > 700+, and almost kill the server.  Ever since then, I don't see the CMS
> > ever stops, even when the load goes down to 50 users.
> >
> > I cut off the logs in between into 3 portion so that I can attach it.
>  :D
> >
> >
> > Thanks,
> > TK
> >
> >
> > On 3/27/08, *Jon Masamitsu* <Jon.Masamitsu at sun.com
>  > <mailto:Jon.Masamitsu at sun.com>> wrote:
> >
> >     Using CMS sometimes needs some tuning (especially
> >     with the 1.4.2 jdk).  Do you have any gc logs
> >     (-XX:+PrintGCDetails) so we can see what's happening?
> >
> >
> >     T.K wrote:
> >      > Hi All,
> >      > We got 5 Sun Web Servers running on Java 1.4.2, and used to use
> the
> >      > default GC for Tenured space. The problem with that is that it
> takes
> >      > 60-80 seconds everytime the GC happens, and the latency on the
> >     site goes
> >      > crazy. So we decided to change it to use the Concurrent Mark
> Sweep
> >      > Collector on one server to test it out. Here's the setting:
> >      >
> >      > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
> >     -XX:NewSize=384M
> >      > -XX:MaxNewSize=384M -XX:PermSize=64M -XX:MaxPermSize=64M
> >      > -XX:CMSInitiatingOccupancyFraction=60
> >      >
> >      > With that setting, the server runs great. But eventually, when
> the
> >      > server reach a medium load (around 100-200 users), the tenured
> >     space is
> >      > always around half full, and the CMS collector starts to run
> >      > continuously one after another. It doesn't hurt the application
> >     for now,
> >      > but it's taking 25% of processing time (we got 4 cpu, so one web
> >     server
> >      > always keep 1 cpu power). I don't see that much cpu utilization
> >     on other
> >      > web server that don't have CMS, and they have more users than the
> one
> >      > with CMS. If we got CMS on all 5 web servers, I'm wondering if
> >     that will
> >      > crash the server or not.  What should I do to decrease the
> processor
> >      > utilization caused by GC?
> >      >
> >      > Also, I'm thinking to use i-CMS on the JVM, and maybe that might
> slow
> >      > down the CMS and reduce the amount of CPU utilization by CMS. Any
> >     thought?
> >      >
> >      > Thanks,
> >      > TK
> >      >
> >      >
> >      >
> >
> ------------------------------------------------------------------------
> >      >
> >      > _______________________________________________
> >      > hotspot-gc-use mailing list
> >      > hotspot-gc-use at openjdk.java.net
> >     <mailto:hotspot-gc-use at openjdk.java.net>
>  >      > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > hotspot-gc-use mailing list
> > hotspot-gc-use at openjdk.java.net
> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080327/94d1c60f/attachment.html 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: oldGclog.txt
Url: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080327/94d1c60f/attachment.txt