Erratic(?) CMS behaviour every 5d

[Bernd Eckenfels]

Hello,


On Fri, Sep 28, 2012 at 6:11 PM, Srinivas Ramakrishna <ysr1729 at gmail.com> wrote:
> ah i see that you do have class unloading enabled.

Yes, customer is using the following options:

-Xverify:none -Xms48G -Xmx48G -Djava.awt.headless=true
-Djava.net.preferIPv4Stack=true -server -d64 -XX:NewRatio=1
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-XX:+ExplicitGCInvokesConcurrent -XX:+CMSClassUnloadingEnabled
-XX:PermSize=100M -XX:MaxPermSize=300M
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-XX:+HeapDumpOnOutOfMemoryError -Xloggc:gc.log -XX:+PrintGCDetails
-XX:-TraceClassUnloading

> But check the size of the
> perm gen (which unfortunately is not reported in the CMS/GC logs unless it
> bails to STW mark-compact).

I used "jmap -heap" while the system was slow, it was in the original
mail. It used 227 of 300mb (75%). Do you know what will be the trigger
for starting CMS when PermGen becomes full? This is a (JBoss)
Appserver, so there are some Reflection objects and multiple unused
classloaders laying around.

> I believe there's also a flag that makes CMS chatty and
> tell you why it's starting a CMS collection.

I added now CMSStatistics=2, will see whats in there next week. Is
that the flag you mean?

The jstat utility has a -gccause option, do you think that will help
me as soon as the system gets into the trashing mode?

> You can also force it to start
> a collection only when the old gen occupancy has exceeded a limit by use of
> -XX:CMSInitiationOccupancy=...  -XX:+UseCMSInitiatingOccupancyOnly, which
> will make it ignore such things as the rate at which the old generation is
> filling up etc.  It could be a bug in that algorithm perhaps.

This was one of my next plans. My problem with that is, that it might
not cover the need for perm collections or a slowly filling old
generation can suddenly get over the (low) occupancy threshhold. But I
gues I have to take the risk and monitor it more closely with that
setting.

> Azeem, I did not look at the GC logs, but the snippets pasted do not show
> concurrent mode failures.

I think there are none (does not contain the substring "fail" :) Only
problem is that nearly all precleans are aborted and the remark phases
are very long.

Thans for looking into this, all your ideas are great help for me.

Bernd