Free space calculation of heap

Sun Jun 3 23:16:30 PDT 2012

Hi Srinivas,

Thanks for your response, see my comments in line,

Regards,
Bond

>>> Srinivas Ramakrishna <ysr1729 at gmail.com> 6/2/2012 3:10 AM >>>
Hi Bond -- some partial answers and comments inline below:-

On Fri, Jun 1, 2012 at 2:48 AM, Bond Chen <Bond.Chen at lombardrisk.com> wrote:

> Hi,
>
> Our application have suffer a bad fragmentation issue at production env, I
> need switch on lots of parameter to analysis the gc, but I'm not fully
> understand the all of the output.
>
> By adding parameter -XX:PrintFLSStatistics=2 , The BinaryTreeDictionary
> and  IndexedFreeLists of old gen and perm gen have been printed out
>
> q1: the total free space + used space of old gen calculation from gc
> output is 128KB less than actual old gen capacity
>

Can you elaborate by pointing out the specific line(s) at which you are
seeing this issue? (For example, are you adding the free space reported in
the BTD and the IFL,
and the used space of old gen reported in the PrintHeapAtGC output, and
comparing that to the Old Gen capacity reported there and finding the
difference?)

I'll reproduce those numbers below:-

Free in BTD: 546527717
Free in IFL:       1221215
                     -----------------
Total Free:    547748932  heap words =  (547748932 * 8 /1024) KB = 4279289
KB

As reported in PrintHeapAtGC: total 4476928K, used 197511K
>From above line, computed free space = (4476928 - 197511) KB = 4279417 KB

"Lost" free space = (4279417 - 4279289) KB = 128 KB

OK, given that i arrived at the number you reported as missing, I guess I
understood yr question correctly.
[BOND] YES, THAT'S HOW I CALC THE FREE SPACE.

This is interesting. It's been a great while since I looked at this code,
but I know that CMS uses
part of the old gen space to store certain book-keeping structures
associated
with promotion ("promo info" and certain header word "spooling buffers") in
the old gen heap. It allocates these
buffers out of the old gen eagerly and doesn't return them unless we
completely run out of space. (This policy
should probably be revised, so it lazily returns free buffers.) However, it
was my understanding that these
buffers are reported as part of the used space. May be the book-keeping for
space used doesn't take these
into account, in which case I am surprised. Probably worth poking around in
the code to see what's going on.
I'll try and do that if I get some time, unless someone from the JVM team
gets to it sooner, especially
those who may have poked around in that code recently may be...

> q2: all counter of the BinaryTreeDictionary and  IndexedFreeLists of perm
> gen is zero, why?
>

The perm gen allocation patterns were sufficiently simple that a linearly
allocated single slab of space
was used instead of free lists, with the idea that the "holes" from freeing
up of class objects in the perm
gen would not reclaim much usable space. This is probably worth revising
with all of the perm gen objects
that will be constantly created and reclaimed with dynamic languages
running atop the JVM. Jon may
recall the details of why it was done that way. Much of this will anyway
change under the new perm gen
allocation regime, where the perm gen has been moved out of the "Java heap"
into the "native heap", so
probably not worth worrying about any more.

>
> q3: what's the meaning of " frag=0.0045", fragmentation ratio? and can
> someone provide the formula?
>

It's a normalized number between 0 and 1, with 1 indicating "maximal
fragmentation" and 0 indicating no
fragmentation. It's computed as

  1 -   (Sum(b_1^2)/(Sum(b_i))^2)
[BOND]: (Sum(b_1^2)/(Sum(b_i))^2) , THE FIRST ONE IS b_i OR b_1 (one) ?

where the sum is over all free blocks i, with block i of size b_i .

>
> q4: which parameter enables the "output log2 of gc log"? how to interpret
> the output
>

The output lists, for each free list size in the "small size" range (i.e.
block sizes through 256 heapwords),
the demand-rate for that block size, lists the old, and current rate, then
based on that estimates how many
blocks it needs to keep for that size. The sweep then makes coalescing
decisions
based on the estimated demand for the blocks under question.

I believe it's produced from PrintFLSCensus or something like that?
[BOND]: MY OPTION DIDN'T INCLUDE PrintFLSCensus, SHOULD BE ONE OF MY NEW ADDED OPTIONS THIS TIME
PrintFLSStatistics=2 (CHANGE =1 to =2)
PrintPromotionFailure
PrintHeapAtGC

-- ramki

> /**output log1 of gc log*/
>
> 67.016: [ParNew
> Desired survivor size 111411200 bytes, new threshold 2 (max 32)
> - age   1:   42577240 bytes,   42577240 total
> - age   2:   74417368 bytes,  116994608 total
> : 1512523K->153985K(1523200K), 0.2866570 secs]
> 1649586K->351496K(6000128K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 546527717
> Max   Chunk Size: 546527717
> Number of Blocks: 1
> Av.  Block  Size: 546527717
> Tree      Height: 1
> Statistics for IndexedFreeLists:
> --------------------------------
> Total Free Space: 1221215
> Max   Chunk Size: 256
> Number of Blocks: 39302
> Av.  Block  Size: 31
>  free=547748932 frag=0.0045
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> Statistics for IndexedFreeLists:
> --------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
>  free=0 frag=0.0000
> , 0.2868167 secs] [Times: user=0.91 sys=0.13, real=0.29 secs]
> Heap after GC invocations=6 (full 1):
>  par new generation   total 1523200K, used 153985K [0xfffffd7e5ec10000,
> 0xfffffd7ec9010000, 0xfffffd7ec9010000)
>  eden space 1305600K,   0% used [0xfffffd7e5ec10000, 0xfffffd7e5ec10000,
> 0xfffffd7eae710000)
>  from space 217600K,  70% used [0xfffffd7eae710000, 0xfffffd7eb7d70438,
> 0xfffffd7ebbb90000)
>  to   space 217600K,   0% used [0xfffffd7ebbb90000, 0xfffffd7ebbb90000,
> 0xfffffd7ec9010000)
>  concurrent mark-sweep generation total 4476928K, used 197511K
> [0xfffffd7ec9010000, 0xfffffd7fda410000, 0xfffffd7fda410000)
>  concurrent-mark-sweep perm gen total 524288K, used 238680K
> [0xfffffd7fda410000, 0xfffffd7ffa410000, 0xfffffd7ffa410000)
> }
>
>
> /***/
>
> /**output log2 of gc log**/
> 2012-05-31T15:23:18.293+0800: 9.731: [CMS-concurrent-sweep-start]
> size[1] : demand: 0, old_rate: 0.000000, current_rate: 0.000000, new_rate:
> 0.000000, old_desired: 0, new_desired: 0
> size[2] : demand: 0, old_rate: 0.000000, current_rate: 0.000000, new_rate:
> 0.000000, old_desired: 0, new_desired: 0
> size[3] : demand: 0, old_rate: 0.000000, current_rate: 0.000000, new_rate:
> 0.000000, old_desired: 0, new_desired: 0
> size[4] : demand: 0, old_rate: 0.000000, current_rate: 0.000000, new_rate:
> 0.000000, old_desired: 0, new_desired: 0
> size[5] : demand: 0, old_rate: 0.000000, current_rate: 0.000000, new_rate:
> 0.000000, old_desired: 0, new_desired: 0
> ...............................
> size[250] : demand: 0, old_rate: 0.000000, current_rate: 0.000000,
> new_rate: 0.000000, old_desired: 0, new_desired: 0
> size[251] : demand: 0, old_rate: 0.000000, current_rate: 0.000000,
> new_rate: 0.000000, old_desired: 0, new_desired: 0
> size[252] : demand: 0, old_rate: 0.000000, current_rate: 0.000000,
> new_rate: 0.000000, old_desired: 0, new_desired: 0
> size[253] : demand: 0, old_rate: 0.000000, current_rate: 0.000000,
> new_rate: 0.000000, old_desired: 0, new_desired: 0
> size[254] : demand: 0, old_rate: 0.000000, current_rate: 0.000000,
> new_rate: 0.000000, old_desired: 0, new_desired: 0
> size[255] : demand: 0, old_rate: 0.000000, current_rate: 0.000000,
> new_rate: 0.000000, old_desired: 0, new_desired: 0
> size[256] : demand: 0, old_rate: 0.000000, current_rate: 0.000000,
> new_rate: 0.000000, old_desired: 0, new_desired: 0
> demand: 1, old_rate: 0.000000, current_rate: 0.103071, new_rate: 0.103071,
> old_desired: 0, new_desired: 0
>
> ***/
>
> Many Thanks
> Bond
>
>
> This e-mail together with any attachments (the "Message") is confidential
> and may contain privileged information. If you are not the intended
> recipient (or have received this e-mail in error) please notify the sender
> immediately and delete this Message from your system.  Any unauthorized
> copying, disclosure, distribution or use of this Message is strictly
> forbidden.
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net 
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use 
>

This e-mail together with any attachments (the "Message") is confidential and may contain privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this Message from your system.  Any unauthorized copying, disclosure, distribution or use of this Message is strictly forbidden.