Excessive "GC" memory area size [was: Re: Determining "GC" memory area size]

[Zhengyu Gu]

Hi Glyn,

On 12/13/2017 09:35 AM, Glyn Normington wrote:
> Thanks Zhengyu! That got me quite a bit further.
> 
> The NMT virtual memory map showed the mmap portion of GC space 
> consisting of:
> * the two marking bitmaps you mentioned previously, each 1/64 of heap size.
> * a mark stack of 32 MB which is 8* MarkStackSize (but which could grow 
> to 8* MarkStackSizeMax or 128 MB).
> * three 700 KB areas, although I'm not sure what these are used for or 
> how to predict their total size.
> 
> This leaves the malloc portion of GC space "17615KB #2312" which I think 
> was due 2312 calls to malloc. Again, not sure what these are used for or 
> how to predict their total size except that I presume they include the 
> task queues you mentioned previously.

Can you try on JDK10? it has enhancement [1] which can show what memory 
allocation is for, on each allocation site.


[1] https://bugs.openjdk.java.net/browse/JDK-8139673

Thanks,

-Zhengyu

> 
> Outstanding questions:
> 1. Is the mmap space which consumed three 700 KB areas above fixed, 
> proportional to heap size, or what?
> 2. How many task queues are there for a given number of GC threads?
> 3. Task queues can overflow, so what is the maximum space each one can 
> occupy?
> 4. Of the malloc space not consumed by task queues, how much is fixed 
> and how much proportional to heap size, number of GC threads, etc.?
> 
> The aim is to calculate a reasonable estimate of an upper bound for GC 
> space consumption.
> 
> Raw data for anyone who's interested: 
> https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6
> 
> Regards,
> Glyn
> 
> On Wed, Dec 13, 2017 at 12:27 PM, Zhengyu Gu <zgu at redhat.com 
> <mailto:zgu at redhat.com>> wrote:
> 
>     Hi Glyn,
> 
>     You can run NMT with detail tracking
>     (-XX:NativeMemoryTracking=detail) then use jcmd (jcmd <pid>
>     VM.native_memory detail) to find out all allocations for GC.
> 
>     For example:
> 
>     [0x00007f9fba000000 - 0x00007f9fbc000000] reserved 32768KB for *GC* from
>          [0x00007fa0130260ef] G1CMMarkStack::resize(unsigned long)+0x8f
>          [0x00007fa01302cf62]
>     G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>     G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>          [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
>          [0x00007fa013ca26af] universe_init()+0x6f
> 
>              [0x00007f9fba000000 - 0x00007f9fbc000000] committed 32768KB
>     from
>                  [0x00007fa01302610d] G1CMMarkStack::resize(unsigned
>     long)+0xad
>                  [0x00007fa01302cf62]
>     G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>     G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>                  [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
>                  [0x00007fa013ca26af] universe_init()+0x6f
> 
> 
>     [0x00007fa012ac5395] ArrayAllocator<unsigned
>     long>::allocate(unsigned long, MemoryType)+0x245
>     [0x00007fa012ac5448] unsigned long*
>     BitMap::reallocate<CHeapBitMapAllocator>(CHeapBitMapAllocator
>     const&, unsigned long*, unsigned long, unsigned long)+0x38
>     [0x00007fa012ac509f] CHeapBitMap::initialize(unsigned long)+0x6f
>     [0x00007fa013093b9c]
>     G1PageBasedVirtualSpace::G1PageBasedVirtualSpace(ReservedSpace,
>     unsigned long, unsigned long)+0x13c
>                                   (malloc=129KB *type=GC* #6)
> 
>     Thanks,
> 
>     -Zhengyu
> 
> 
> 
> 
>     On 12/13/2017 04:39 AM, Glyn Normington wrote:
> 
>         On the basis of the reply below, we have been thinking of
>         estimating the
>         size of the "GC" memory area as a little more than (1/32) *
>         heapSize.
>         However, running `java -version` with a 350m heap results in
>         "GC" memory as
>         reported by NMT of over 17% of the heap size, so this estimate
>         is way out.
> 
>         If the analysis below is accurate, then given the number of GC
>         threads is
>         10 (ParallelGCThreads=8, ConcGCThreads=2), can there be many more
>         taskqueues than GC threads?
> 
>         Or is there another explanation for the additional "GC" size?
> 
>         Thanks,
>         Glyn
> 
>         [1]
>         https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>
> 
>         On Fri, Nov 10, 2017 at 7:05 PM, Zhengyu Gu <zgu at redhat.com
>         <mailto:zgu at redhat.com>> wrote:
> 
>             Hi Glyn,
> 
> 
>                 Can anyone here tell me how the GC memory area size is
>                 determined? If
>                 there
>                 is documentation, so much the better as we'd prefer not
>                 to depend on code
>                 details that might flux arbitrarily.
> 
> 
>             GC memory is mainly data structures used by GC runtime. It
>             can be varied
>             by collector used, size of the Java heap, the number of GC
>             threads and etc.
>             and, of course, the application itself.
> 
>             Some are *fixed* costs, which can be estimated. E.g. two
>             marking bitmaps
>             used by G1, each costs 1/64 of heap size (assuming default
>             object
>             alignment).
> 
>             Some are *semi-fixed*, e.g. taskqueue's fixed cost is about
>             1M for each
>             queue on 64-bits VM, but it can overflow. And the number of
>             task queues is
>             proportional to the number of GC threads.
> 
>             Then there are factors from application itself,  such as
>             object mutation
>             rate, inter-generation/region references, etc.
> 
> 
>             I don't see a single formula for estimating GC memory size.
>             If you are
>             using G1, the biggest overhead comes from 2 bitmaps (1/32 *
>             heap size).
> 
>             Thanks,
> 
>             -Zhengyu
> 
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> Regards,
> Glyn