metaspace proportion of fragmentation

[Brian Toal]

Thanks for the reply Thomas.

We are getting a OOME on metaspace and do not set
CompressedClassSpaceSize.  It seems our used CompressedClassSpaceSize is
~100MB.  If this isn't set, will it use as much of the specified
XX:MaxMetaspaceSize
as needed?

We do have a lot of class loaders, after looking closely at the heap.
Looks like ~11k loaders, where the majority of them are
sun.reflect.DelegatingClassLoader's corresponding to reflective method
instances that are being strongly referenced.  I'm not sure if the chunk
size is 64k because that would lead to ~687MB of Metaspace going to the
initial allocation for each loader, however looking at the output of "jcmd
<pid> GC.class_stats" and summing up the total for all
DelegatingClassLoader's it shows only ~46MB is used and the remaining
classes account for ~840MB.  Maybe the total accumulated chunk memory is
not part of the output of "jcmd <pid> GC.class_stats", do you know if
allocated by unused space by the loader is reported here?  If not is there
anyway to get this info on a production JVM?

Looking at the pdf was on the reference of 2, it seem like the
DelegatingClassLoader's are consuming 4 small chunks of 512 words which is
4 x 512 words x 8 bytes/word so as a lower bound i would expect the cost of
all those loaders to be ~171MB.  The metaspace usage in the JVM is ~940MB
and compressed class space is ~100mb, so adding the ~171MB it seems to
bring us to a total of ~1.18G which is roughly the value of
XX:MaxMetaspaceSize.

I've found a few ways to limit the number of DelegatingClassLoader's by
either changing the inflation threshold or we could possibly just break the
path back to gc root of the class instance that is owned by the
corresponding DelegatingClassLoader.  It's a shame that the same
DelegatingClassLoader isn't reused, but I suppose the finer granularity of
method to classloader is to increase the chances that the loader can be
unloaded when the method is not longer referenced.

On Tue, Oct 25, 2016 at 2:37 AM, Thomas Stüfe <thomas.stuefe at gmail.com>
wrote:

> Hi Brian,
>
> On Sat, Oct 22, 2016 at 1:55 AM, Brian Toal <brian.toal at gmail.com> wrote:
>
>> Good evening.  In a application that I'm responsible for, Metaspace is
>> set to 1.1GB.  Specifically the following flags are set:
>>
>> -XX:MetaspaceSize=1152m
>> -XX:MaxMetaspaceSize=1152m
>> -XX:MinMetaspaceFreeRatio=0
>> -XX:MaxMetaspaceFreeRatio=100
>>
>> However we are getting a OOME when metaspace size hits 80% of 1.1GB.
>>
>
> Out of metaspace or out of compressed class space? If the latter, have you
> set CompressedClassSpaceSize?
>
>
>> Doing a bit or research it seems that Metaspace is known to fragement the
>> memory when a loader needs to acquire memory from the current chunk, and
>> the current chuck can't accomodate the request, the pointer is bumped to
>> the next available chunk, meaning any free memory in the previous chunks
>> block is gone with the wind.
>>
>
> No. The remaining space is put into freelists (both on chunk and on block
> level) and used for follow-up requests, should the size fit. In our
> experience, we see very little wastage due to "half-eaten blocks/chunks.
>
> There are other possible waste scenarios:
>
> 1) you have a lot of class loaders living in parallel. Each one will take
> a chunk of memory (its current chunk) and satisfy memory requests from
> there. This means that the current chunk always contains a portion of
> still-unused memory which may be used by this class loader in the future
> but already counts against MaxMetaspaceSize. However, to make this hurt,
> you really have to have very many class loaders in parallel, as the maximum
> possible overhead for this scenario cannot exceed the size of a medium
> chunk per classloader (64k?)
>
> 2) The scenario described in this JEP here: https://bugs.openjdk.
> java.net/browse/JDK-8166690 .
>
> 3) real fragmentation (i.e. a mixture of in-use and free chunks).
>
> In my practice, I keep seeing (2). Hence the JEP, which will hopefully
> help.
>
> Kind Regards, Thomas
>
>
>
>> More than happy if someone corrects my understand here or can point me to
>> a good reference that explains this in detail.
>>
>> My question is, how to do I monitor current usage + fragmentation so the
>> proportion of free space can be monitored?
>>
>> Also is there any tuning that can take place to reduce the proportion of
>> fragmentation?
>>
>> Does compressed cache acquire memory from memory set aside from memory
>> allocated via MaxMetaspaceSize?
>>
>> Thanks in advance.
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20161025/49cea8ca/attachment.html>