How to alert for heap fragmentation

[Srinivas Ramakrishna]

In the absence of fragmentation, one would normally expect the max chunk
size of the CMS generation
to stabilize at some reasonable value, say after some 10's of CMS GC
cycles. If it doesn't, you should try
and use a larger heap, or otherwise reshape the heap to reduce promotion
rates. In my experience,
CMS seems to work best if its "duty cycle" is of the order of 1-2 %, i.e.
there are 50 to 100 times more
scavenges during the interval that it's not running vs the interva during
which it is running.

Have Nagios grep the GC log file w/PrintFLSStatistics=2 for the string
"Max  Chunk Size:" and pick the
numeric component of every (4n+1)th match. The max chunk size will
typically cycle within a small band,
once it has stabilized, returning always to a high value following a CMS
cycle's completion. If the upper envelope
of this keeps steadily declining over some 10's of CMS GC cycles, then you
are probably seeing fragmentation
that will eventually succumb to fragmentation.

You can probably calibrate a threshold for the upper envelope so that if it
falls below that threshold you will
be alerted by Nagios that a closer look is in order.

At least something along those lines should work. The toughest part is
designing your "filter" to detect the
fall in the upper envelope. You will probably want to plot the metric, then
see what kind of filter will detect
the condition.... Sorry this isn't much concrete help, but hopefully it
gives you some ideas to work in
the right direction...

-- ramki

On Thu, Oct 11, 2012 at 4:27 PM, roz dev <rozdev29 at gmail.com> wrote:

> Hi All
>
> I am using Java 6u23, with CMS GC. I see that sometime Application gets
> paused for longer time because of excessive heap fragmentation.
>
> I have enabled PrintFLSStatistics flag and following is the log
>
>
> 2012-10-09T15:38:44.724-0400: 52404.306: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: -668151027
> Max   Chunk Size: 1976112973
> Number of Blocks: 175445
> Av.  Block  Size: 20672
> Tree      Height: 78
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 10926
> Max   Chunk Size: 1660
> Number of Blocks: 22
> Av.  Block  Size: 496
> Tree      Height: 7
>
>
> I would like to know from people about the way they track Heap
> Fragmentation and how do we alert for this situation?
>
> We use Nagios and I am wondering if there is a way to parse these logs and
> know the max chunk size so that we can alert for it.
>
> Any inputs are welcome.
>
> -Saroj
>
>
>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20121011/f6d365ea/attachment.html