jdk 8u40 - MaxMetaspaceSize and CompressedClassSpaceSize

[Srinivas Ramakrishna]

Hi Vitaly --

The allocated (but not necessarily touched/faulted) memory reported above
(by the jvm) is the one corresponding to "committed". The "reservation"
reported above is only of the virtual *address space* within the process,
not virtual *memory*  (i.e. no swap reservations or the like). I didn't
know that ulimits could prevent virtual address space (_not_ virtual
memory) reservation.

I think your question is valid, in that looking at the VA reservation, one
feels that the JVM might eventually decide to commit beyond the max space
you asked for in your command-line spec. At the very least it is confusing,
and someone who knows why could explain the reason for the larger than
user-requested VA space (not VA memory yet in yr snippet) reservation when
it won't be used  -- if it won't be used, and here I am giving the benefit
of the doubt to the JVM in that it'll never try to commit those pages
despite having reserved that in its virtual address space.

-- ramki

On Tue, Apr 7, 2015 at 5:01 PM, Vitaly Davidovich <vitalyd at gmail.com> wrote:

> By the way, forgot to mention that while doing some searching around this,
> saw this: https://issues.apache.org/jira/browse/HADOOP-11364.  So anyone
> with multi-tenant servers that impose mem limits may hit this.  But again,
> I'm trying to understand the flags involved and not saying this is a big
> problem.
>
> sent from my phone
> On Apr 7, 2015 7:55 PM, "Vitaly Davidovich" <vitalyd at gmail.com> wrote:
>
>> Hi Ramki,
>>
>> I agree in principle that a 1GB reservation on 64bit isn't an issue in
>> and of itself.  The problem comes up, however, if a program is run with
>> ulimit on VA and someone sized their java app appropriately in, say, java
>> 7.  With java 8, there can be regressions.
>>
>> Second, if you disable swap and overcommit on linux, larger virtual mem
>> allocations may fail.
>>
>> Having said all that, I'm curious whether there's a way to restrict this
>> with 1 flag in java 8 and whether the linked documentation is correct or
>> not.
>>
>> Thanks
>>
>> sent from my phone
>> On Apr 7, 2015 7:26 PM, "Srinivas Ramakrishna" <ysr1729 at gmail.com> wrote:
>>
>>> Hi Vitaly --
>>>
>>> Because reserved but unused virtual address space is, for all practical
>>> purposes today, irrelevant for 64 bit processes,
>>> a question to ask is what you see when you do this for 32-bit processes
>>> (where of course the oop compression
>>> comments wouldn't of course apply, but the amount of VA space used would
>>> be much more relevant and parsimony
>>> -- or lack thereof -- in VA space reservation would visibly affect the
>>> bottom-line in a 32-bit VA space).
>>>
>>> -- ramki
>>>
>>> On Tue, Apr 7, 2015 at 12:27 PM, Vitaly Davidovich <vitalyd at gmail.com>
>>> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> It seems that putting a cap on MaxMetaspaceSize has no impact on amount
>>>> of
>>>> memory reserved *unless* CompressedClassSpaceSize is also set.  For
>>>> example,
>>>>
>>>> java -Xmx64m -XX:MaxMetaspaceSize=16m Test
>>>>
>>>> yields the following:
>>>>
>>>>  Metaspace       used 2421K, capacity 4486K, committed 4864K, reserved
>>>> 1056768K
>>>>   class space    used 261K, capacity 386K, committed 512K, reserved
>>>> 1048576K
>>>>
>>>> Adding -XX:CompressedClassSpaceSize=16m yields:
>>>>
>>>> Metaspace       used 2421K, capacity 4486K, committed 4864K, reserved
>>>> 24576K
>>>>   class space    used 261K, capacity 386K, committed 512K, reserved
>>>> 16384K
>>>>
>>>> According to
>>>>
>>>> http://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.html
>>>> :
>>>>
>>>> >
>>>> > If UseCompressedOops is turned on and UseCompressedClassesPointers is
>>>> > used, then two logically different areas of native memory are used for
>>>> > class metadata. UseCompressedClassPointers uses a 32-bit offset to
>>>> > represent the class pointer in a 64-bit process as does
>>>> UseCompressedOops for
>>>> > Java object references. A region is allocated for these compressed
>>>> class
>>>> > pointers (the 32-bit offsets). The size of the region can be set with
>>>> > CompressedClassSpaceSize and is 1 gigabyte (GB) by default. The space
>>>> for
>>>> > the compressed class pointers is reserved as space allocated by mmap
>>>> at
>>>> > initialization and committed as needed. The MaxMetaspaceSize applies
>>>> to
>>>> > the sum of the committed compressed class space and the space for the
>>>> other
>>>> > class metadata.
>>>>
>>>>
>>>> Unless I'm misunderstanding, the last sentence there seems to imply that
>>>> MaxMetaspaceSize would put a cap on the compressed class space as
>>>> well.  Or
>>>> is the "committed" part of the last sentence the key? There's no way to
>>>> cap
>>>> the *reserved* amount with 1 flag? Basically, what's the java 8
>>>> equivalent
>>>> to MaxPermSize=X in java 7 and earlier?
>>>>
>>>> The reason this came up is because java 8 seems to have a much higher
>>>> virtual mem reserve charge than java 7, even for an app with a simple
>>>> main
>>>> that does nothing, and metaspace/compressed class space reservation
>>>> appears
>>>> to be the biggest contributor.
>>>>
>>>> Thanks
>>>>
>>>
>>>