CMS collection keep working during holiday

[Y Srinivas Ramakrishna]

Hi Ken -- Good to know that your main problem was solved.

> Thanks to all of you. It seems work after adding
> -XX:+CMSClassUnloadingEnabled as you suggested.
> 
> So, why classloader create so many HashMap entries and Vector 
> elements? It
> seems that only a few people have this problem.

I suspect that if you used jconsole to track the #classes loaded,
it would likely show an unbounded upward trend (or at least
an upward trend that would correlate with the slope of the lower
envelope of the old generation occupancy).

The vectors are class loaders that your application container(s)
use to load classes. The problem with CMS's default of not concurrently
collecting classes here is that the classes which live in the perm gen
point to their class loaders which live in the regular heap (old gen,
eventually). So the class loaders do not get collected either. Further,
the uncollected classes may have heavy-weight static fields associated
with them (data which again lives in the regular heap). Thus what might
be a few small classes that we schlep around in our perm gen, might be
holding onto a lot of garbage that collects in the old gen.

That's what we saw in your application.

It so happens that in some cases you can actually clearly see the
deleterious effect of accumulating perm gen garbage on the performance
of not only CMS (which, as in your case, had an artifically bloated
footprint and was running useless cycles), but on scavenges --
the latter is two-fold: firstly the quasi-immortal (modulo
mark-compact which reclaims them) class loaders and other class-cohorts
act as barriers to coalition and tend to fragment the old gen, causing
much pain to scavenges, which slow down over time; further,
the dead classes in the perm gen act as "dead roots" that inflates
the root-scanning time for scavenges and further slows down scavenges.
In a recent exchange, i think on this list, we saw a nice expample
of that. We can probably extract a time-series plot of the scavenge
times from your logs to see if we see a similar trending which
is often a signature of problems such as these. (Of course similar
signatures would appear also with some memory leaks, as Tony and Kirk
suspected earlier.)

Now a word about the default setting of CMSClassUnloadingEnabled. This setting
was selected a number of years ago when it was relatively uncommon to
have applications running in containers. In such cases, one would
expect a small upper bound on the number of classes an application
would load and thus it did not hurt much to not unload unreachable
classes, thus avoid the extra work that CMS would need to do especially
during the stop-world remark pause. That default setting however appears
to be the wrong setting for the way a large majority of the applications
are run today inside containers which causes a proliferation of classes
and loaders.

We have had a bug [1] filed for flipping the switch for a while but have not
bitten the bullet and either take the resulting hit on CMS remark times
(or appropriately deal with it; [2] is only one of several possibilities
here). I think we'll address [1] soon, so that most users (such as Ken
 would not have to worry about this setting. Instead, power users
who know what they are doing andcan compute a sufficiently small
and tight upper bound on the total number and size of classes
(and cohorts) they load during the life of their application
would be required to throw off the switch explicitly.
In my own recent (and admittedly somewhat biased) experience,
the fraction of applications for which a non-trivial upper bound
exists has been shrinking over time as more and more applications
deploy on Java EE / application servers / containers.

Comments, votes, plebiscite?

-- ramki

[1] 6329603 CMS: +CMSClassUnloadingEnabled and +CMSPermGenSweepingEnabled by default
[2] 6543076 CMS: Adaptive collection of perm gen


> 
> vm options:
> 
> -server -showversion -Djava.security.policy=.\java.policy -Xms1536M
> -Xmx1536M -XX:NewSize=256M -XX:+PrintGCDetails -XX:+PrintGCTaskTimeStamps
> -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -Xloggc:gc.log
> -XX:+PrintReferenceGC -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:SurvivorRatio=4 -XX:CMSInitiatingOccupancyFraction=55
> -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
> -XX:MaxTenuringThreshold=1 -XX:SoftRefLRUPolicyMSPerMB=0
> -Dcom.sun.management.jmxremote.port=18002
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> 
> 
> gc log:
> http://www.nabble.com/file/p19912471/gc.log gc.log 
> 
> jconsole:
> http://www.nabble.com/file/p19912471/jconsole_20081010.jpg 
> 
> 
> 
> 
> > Y Srinivas Ramakrishna wrote:
> > > The mention of class loaders below reminds me to remind you that
> > > CMS by default will not collect class objects. To force CMS
> > > to collect classes in the perm gen, you would want to
> > > -XX:+CMSClassUnloadingEnabled. Try to see if that made any
> > > difference to the apparent inability of CMS to collect
> > > those apparently otherwise unreachable objects.
> > >
> > > -- ramki
> > >
> > > ----- Original Message -----
> > > From: "Ken-- at newsgroupstats.hk" <dragonken at gmail.com>
> > > Date: Thursday, October 9, 2008 3:10 am
> > > Subject: Re: CMS collection keep working during holiday
> > > To: hotspot-gc-dev at openjdk.java.net
> > >
> > >
> 
> -- 
> View this message in context: http://www.nabble.com/CMS-collection-keep-working-during-holiday-tp19773575p19912471.html
> Sent from the OpenJDK Hotspot Garbage Collection mailing list archive 
> at Nabble.com.
>