jdk 8u40 - MaxMetaspaceSize and CompressedClassSpaceSize

Wed Apr 8 09:23:36 UTC 2015

Hi Vitaly,

On 2015-04-08 04:26, Vitaly Davidovich wrote:
> Thanks Ramki, that does clear it up a bit.  So you're saying the JVM notion
> of reserved is basically (on linux) mmap()'ing anon memory with PROT_NONE
> equivalent - i.e. no commit charge.  Is that right?

That's sounds right to me.

The code path to the mmap:
os::reserve_memory
   os::pd_reserve_memory
     anon_mmap
       mmap( ... PROT_NONE ... , ... MAP_NORESERVE ... )

I did an experiment with some time ago that can be found at:
http://mail.openjdk.java.net/pipermail/hotspot-dev/2013-July/010226.html

Independent of the over-commit settings in the Linux kernel, I could 
still "reserve" 2TB of memory on my 8GB machine, if PROT_NONE was used.

>
> Ok, so the issue with ulimit is probably that it actually restricts address
> space, and not virtual memory.
>
> As for the reservation in Hotspot, why is it reserving that address space
> if it has no intention of using it?

I agree that we shouldn't have to reserve 1 GB of virtual address space 
for the compressed class space when MaxMetaspaceSize is set to  a lower 
value. It should be enough to allocate MaxMetaspaceSize of compressed 
class space. Unfortunately, we don't know how much Metadata will be put 
inside the compressed class space vs the other metaspace memory areas 
(containing the non-class metadata). So, we would probably end up 
allowing ~ 2 x MaxMetaspaceSize of memory to be reserved if the program 
committed MaxMetaspaceSize of non-class metadata.

We could reserve a contiguous memory area if user has set a low 
MaxMetaspaceSize, say below 1 GB, and use that area for both classes and 
non-class metadata. With that in place the MaxMetaspaceSize flag would 
limit both the reserved and committed memory. Unfortunately, this has a 
draw-back that mmap will destroy the reservation of the memory area if a 
sub-sequent "commit" request would fail. We would have to handle that 
somehow.

>   Or if it does use it, I'm assuming
> it'll change protection on it and then possibly oom at that point (if
> commit charge is over the limit).
>
> But anyway, back to the question at hand.  Is there a single flag that
> would cap the amount of VA requested? It seems odd that MaxMetaspaceSize by
> itself doesn't change VA reservation, but adding CompressedClassSpaceSize
> does.  Moreover, if I'm capping max size, why even reserve more VA in that
> case?

MaxMetaspaceSize is designed to limit the amount of committed memory and 
not to limit the amount of reserved memory. At this moment, there are no 
flag to limit the amount of reserved memory by the metaspaces. If that 
feature is needed, it would have to be implemented.

Thanks,
StefanK

>
> Thanks
>
> sent from my phone
> On Apr 7, 2015 9:38 PM, "Srinivas Ramakrishna" <ysr1729 at gmail.com> wrote:
>
>> Hi Vitaly --
>>
>> Let me give an example: In a multi-threaded process, if we want some piece
>> of anon memory to be placed at a specific spot in our (i.e. process's) VA
>> address space, we can first "reserve" that address space (tit is in that
>> sense purely local to the process, there are no physical pages backing it,
>> no swap reservation, no OS accounting, nothing). We then can decide at
>> leisure what to do with that address space, knowing that that address space
>> in our process is reserved for us and "no one can take that away from us."
>> This is what the JVM calls an address space reservation.
>>
>> What the JVM calls "commit" is when we reserve the virtual memory with the
>> OS. Committed memory may or may not be backed by physical pages. (For
>> example, on Linux wjere you allow overcommit.) It's only when you access
>> that page will it be faulted in from the set of free pages in RAM (if there
>> are any left and you aren't overcommitted).
>>
>> What is confusing you (and many others) is the use of the term "reserved".
>> Here reserved is used in the sense of reserving the address space in the
>> process. Nothing else. No OS book-keeping to reserve virtual *memory*,
>> nothing.
>>
>> At least that was what it was a couple of years ago, but I haven't looked
>> at the code in recent times (but given this discussion, I'll go and do so
>> soon!).
>> -- ramki
>>
>>
>> On Tue, Apr 7, 2015 at 5:47 PM, Vitaly Davidovich <vitalyd at gmail.com>
>> wrote:
>>
>>> Ramki,
>>>
>>> I'm not sure I fully understand your distinction between virtual address
>>> space vs virtual memory.  If you disable swap and overcommit and you
>>> mmap/brk from the kernel, it has to ensure that the reservation could be
>>> backed up if it were committed; if you take away swap and don't allow linux
>>> to overpromise (i.e. overcommit), the malloc will fail.  See here for what
>>> I'm talking about: http://www.etalabs.net/overcommit.html
>>>
>>> The committed memory is memory that is now backed by physical ram (or
>>> swap), it has either been touched or manually committed - it's not a
>>> reservation (which has no physical backing, just kernel VM accounting
>>> behind it).
>>>
>>> Did I misunderstand you?
>>>
>>> sent from my phone
>>> Hi Vitaly --
>>>
>>> The allocated (but not necessarily touched/faulted) memory reported above
>>> (by the jvm) is the one corresponding to "committed". The "reservation"
>>> reported above is only of the virtual *address space* within the process,
>>> not virtual *memory*  (i.e. no swap reservations or the like). I didn't
>>> know that ulimits could prevent virtual address space (_not_ virtual
>>> memory) reservation.
>>>
>>> I think your question is valid, in that looking at the VA reservation,
>>> one feels that the JVM might eventually decide to commit beyond the max
>>> space you asked for in your command-line spec. At the very least it is
>>> confusing, and someone who knows why could explain the reason for the
>>> larger than user-requested VA space (not VA memory yet in yr snippet)
>>> reservation when it won't be used  -- if it won't be used, and here I am
>>> giving the benefit of the doubt to the JVM in that it'll never try to
>>> commit those pages despite having reserved that in its virtual address
>>> space.
>>>
>>> -- ramki
>>>
>>> On Tue, Apr 7, 2015 at 5:01 PM, Vitaly Davidovich <vitalyd at gmail.com>
>>> wrote:
>>>
>>>> By the way, forgot to mention that while doing some searching around
>>>> this, saw this: https://issues.apache.org/jira/browse/HADOOP-11364.  So
>>>> anyone with multi-tenant servers that impose mem limits may hit this.  But
>>>> again, I'm trying to understand the flags involved and not saying this is a
>>>> big problem.
>>>>
>>>> sent from my phone
>>>> On Apr 7, 2015 7:55 PM, "Vitaly Davidovich" <vitalyd at gmail.com> wrote:
>>>>
>>>>> Hi Ramki,
>>>>>
>>>>> I agree in principle that a 1GB reservation on 64bit isn't an issue in
>>>>> and of itself.  The problem comes up, however, if a program is run with
>>>>> ulimit on VA and someone sized their java app appropriately in, say, java
>>>>> 7.  With java 8, there can be regressions.
>>>>>
>>>>> Second, if you disable swap and overcommit on linux, larger virtual mem
>>>>> allocations may fail.
>>>>>
>>>>> Having said all that, I'm curious whether there's a way to restrict
>>>>> this with 1 flag in java 8 and whether the linked documentation is correct
>>>>> or not.
>>>>>
>>>>> Thanks
>>>>>
>>>>> sent from my phone
>>>>> On Apr 7, 2015 7:26 PM, "Srinivas Ramakrishna" <ysr1729 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Vitaly --
>>>>>>
>>>>>> Because reserved but unused virtual address space is, for all
>>>>>> practical purposes today, irrelevant for 64 bit processes,
>>>>>> a question to ask is what you see when you do this for 32-bit
>>>>>> processes (where of course the oop compression
>>>>>> comments wouldn't of course apply, but the amount of VA space used
>>>>>> would be much more relevant and parsimony
>>>>>> -- or lack thereof -- in VA space reservation would visibly affect the
>>>>>> bottom-line in a 32-bit VA space).
>>>>>>
>>>>>> -- ramki
>>>>>>
>>>>>> On Tue, Apr 7, 2015 at 12:27 PM, Vitaly Davidovich <vitalyd at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi guys,
>>>>>>>
>>>>>>> It seems that putting a cap on MaxMetaspaceSize has no impact on
>>>>>>> amount of
>>>>>>> memory reserved *unless* CompressedClassSpaceSize is also set.  For
>>>>>>> example,
>>>>>>>
>>>>>>> java -Xmx64m -XX:MaxMetaspaceSize=16m Test
>>>>>>>
>>>>>>> yields the following:
>>>>>>>
>>>>>>>   Metaspace       used 2421K, capacity 4486K, committed 4864K, reserved
>>>>>>> 1056768K
>>>>>>>    class space    used 261K, capacity 386K, committed 512K, reserved
>>>>>>> 1048576K
>>>>>>>
>>>>>>> Adding -XX:CompressedClassSpaceSize=16m yields:
>>>>>>>
>>>>>>> Metaspace       used 2421K, capacity 4486K, committed 4864K, reserved
>>>>>>> 24576K
>>>>>>>    class space    used 261K, capacity 386K, committed 512K, reserved
>>>>>>> 16384K
>>>>>>>
>>>>>>> According to
>>>>>>>
>>>>>>> http://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.html
>>>>>>> :
>>>>>>>
>>>>>>>> If UseCompressedOops is turned on and UseCompressedClassesPointers
>>>>>>> is
>>>>>>>> used, then two logically different areas of native memory are used
>>>>>>> for
>>>>>>>> class metadata. UseCompressedClassPointers uses a 32-bit offset to
>>>>>>>> represent the class pointer in a 64-bit process as does
>>>>>>> UseCompressedOops for
>>>>>>>> Java object references. A region is allocated for these compressed
>>>>>>> class
>>>>>>>> pointers (the 32-bit offsets). The size of the region can be set
>>>>>>> with
>>>>>>>> CompressedClassSpaceSize and is 1 gigabyte (GB) by default. The
>>>>>>> space for
>>>>>>>> the compressed class pointers is reserved as space allocated by
>>>>>>> mmap at
>>>>>>>> initialization and committed as needed. The MaxMetaspaceSize
>>>>>>> applies to
>>>>>>>> the sum of the committed compressed class space and the space for
>>>>>>> the other
>>>>>>>> class metadata.
>>>>>>>
>>>>>>> Unless I'm misunderstanding, the last sentence there seems to imply
>>>>>>> that
>>>>>>> MaxMetaspaceSize would put a cap on the compressed class space as
>>>>>>> well.  Or
>>>>>>> is the "committed" part of the last sentence the key? There's no way
>>>>>>> to cap
>>>>>>> the *reserved* amount with 1 flag? Basically, what's the java 8
>>>>>>> equivalent
>>>>>>> to MaxPermSize=X in java 7 and earlier?
>>>>>>>
>>>>>>> The reason this came up is because java 8 seems to have a much higher
>>>>>>> virtual mem reserve charge than java 7, even for an app with a simple
>>>>>>> main
>>>>>>> that does nothing, and metaspace/compressed class space reservation
>>>>>>> appears
>>>>>>> to be the biggest contributor.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>