G1GC, Java8u40ea, Metaspace questions

[Yu Zhang]

Wolfgang,

Please see my comments in line.

Thanks,
Jenny

On 2/13/2015 5:36 AM, Wolfgang Pedot wrote:
> Hello,
>
> my last mail regarding this topic seems to have been lost somewhere but thats not really a problem because my questions have changed:
>
> I am currently evaluation 8u40 early-access (b21 and b23) for an application server which, unfortunately, generates a fair amount of short-lived dynamic classes. So far it runs on 7u67 and has enough PermGen space to survive most days with 1 FullGC to get rid of the classes. This situation is somewhat acceptable but I am really looking forward to not requiring FullGCs for class unloading, hence my tests with 8u40ea.
> I am testing on a smaller system with much less headroom in PermGen and as a first shot I just renamed MaxPermSize to MaxMetaspaceSize and everything looked OK at first, classes where unloaded without FullGCs and I was happy. When the Metaspace baseline-usage grew I noticed that there where quite some FullGCs after all and when looking closer I noticed that they came in pairs:
>
> 276129.049: [Full GC (Metadata GC Threshold) 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-root-region-scan-end, 0.0001449 secs]
> 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-mark-start]
>   492M->476M(904M), 1.5020455 secs]
>     [Eden: 0.0B(336.0M)->0.0B(352.0M) Survivors: 16.0M->0.0B Heap: 492.2M(904.0M)->476.4M(904.0M)], [Metaspace: 183332K->183332K(1247232K)]
>   [Times: user=2.74 sys=0.01, real=1.50 secs]
> 2015-02-05T17:03:06.011+0100: 276130.551: [Full GC (Last ditch collection)  476M->468M(904M), 1.4614149 secs]
>     [Eden: 0.0B(352.0M)->0.0B(360.0M) Survivors: 0.0B->0.0B Heap: 476.4M(904.0M)->468.0M(904.0M)], [Metaspace: 183332K->178688K(1247232K)]
>   [Times: user=2.62 sys=0.01, real=1.46 secs]
>
> MaxMetaspaceSize was set to 228M at the time so it was not completely full yet the JVM threw in a "Last ditch collection" after every (quite frequent) FullGC to desperately try and free some Metaspace.
> I even got to the point where the JVM threw OOME:Metaspace errors even though according to the numbers Metaspace was nowhere near full (different settings though).
MaxMetaspaceSize is the maximum size Metadata space can expand to. GC 
will start with metadata space defined by MetaspaceSize, then 
expand/shrink it.  When it needs to expand it, a 'Metadata GC Threshold' 
full gc is called.  To avoid this kind of full gc, you can increase 
MetaspaceSize.

In jdk8u40, ClassUnloadingWithConcurrentMark(default true) so that the 
classes can be unloaded with concurrent marking.  This will reduce the 
chance of running into Metadata GC Threshold' full gc

>
> After banging my head against various settings regarding Metaspace I decided to run some worst-case tests on my laptop with different MaxMetaspaceSize-values (no other metaspace related settings). My test generates lots of classes while causing only minor increases in old-gen so the JVM runs into FullGC (Metaspace threshold) consistently. I discovered that the reported Metaspace-Size in the gclog (and jconsole) never actually reaches the configured maximum but instead tops out at about 85% of that value. "Commited" will be the Maxsize but "capacity" never goes that high and since my test-system did not have enough headroom to cover that it was collecting because of Metaspace way to often.
>
> This brings me to my first question: What is the reason why the JVM (appearantly) does not use the whole space it is allowed to use?  Is other stuff stored in there but not reported or is there some kind of overhead?
>
> After giving the test-system more headroom it appears to work OK but it always takes a couple of days use before Metaspace-baseline evens out so I will see next week if that did the trick, I dont really want to go "no limit".
>
> Now a more general question:
> I noticed that G1 also triggers a concurrent marking cycle because of "Metaspace threshold", is it using the same threshold that would also trigger a FullGC or is it slightly lower? Is that treshold visible somewhere?
> The reason I am asking is that I want to keep FullGCs down to a minimum so if the class count rises reasonably slow even without causing much oldgen-increase should the concurrent cycle be triggered before the JVM throws in a FullGC most of the time?
You are talking about Full GC caused by 2 reasons:

  *

    Metaspace threshold: we can avoid this my increasing MetadataspaceSize

  * Allocation Failure: usually this happens after a few 'to-space
    exhausted'.  I think this is not what you see in your log.  A lot of
    time this is caused by poor tuning.  The marking cycles are
    triggered after initial marking.  And when initial marking is
    triggered, is decided by InitiatingHeapOccupancyPercent

>
> Quite a long Mail for two questions, any hints would be greatly appreciated!
>
> Wolfgang
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20150213/8459b4e8/attachment.html>