jdk 8u40 - MaxMetaspaceSize and CompressedClassSpaceSize

Wed Apr 8 13:06:45 UTC 2015

Hi Stefan,

Thanks for clarifying some of this.  The one thing I'm still unclear about
is why more VA is reserved when a max committed amount is set to a lower
value.  I'm assuming that if I set MaxMetaspaceSize to, say, 32MB then
trying to use/commit more than that will result in an OOME.  Yet the VA
reservation stays at 1GB.  Or are you saying that 32MB only restricts the
non-class metadata and thus you need to leave additional room for class
metadata? Does MaxMetaspaceSize actually cap both class and non-class
metadata? The doc I linked to earlier suggests it caps both, but if I'm
reading your email right, you seem to be saying that even if
MaxMetaspaceSize is set you still need additional room for class metadata.

Thanks

sent from my phone
On Apr 8, 2015 5:23 AM, "Stefan Karlsson" <stefan.karlsson at oracle.com>
wrote:

> Hi Vitaly,
>
> On 2015-04-08 04:26, Vitaly Davidovich wrote:
>
>> Thanks Ramki, that does clear it up a bit.  So you're saying the JVM
>> notion
>> of reserved is basically (on linux) mmap()'ing anon memory with PROT_NONE
>> equivalent - i.e. no commit charge.  Is that right?
>>
>
> That's sounds right to me.
>
> The code path to the mmap:
> os::reserve_memory
>   os::pd_reserve_memory
>     anon_mmap
>       mmap( ... PROT_NONE ... , ... MAP_NORESERVE ... )
>
> I did an experiment with some time ago that can be found at:
> http://mail.openjdk.java.net/pipermail/hotspot-dev/2013-July/010226.html
>
> Independent of the over-commit settings in the Linux kernel, I could still
> "reserve" 2TB of memory on my 8GB machine, if PROT_NONE was used.
>
>
>> Ok, so the issue with ulimit is probably that it actually restricts
>> address
>> space, and not virtual memory.
>>
>> As for the reservation in Hotspot, why is it reserving that address space
>> if it has no intention of using it?
>>
>
> I agree that we shouldn't have to reserve 1 GB of virtual address space
> for the compressed class space when MaxMetaspaceSize is set to  a lower
> value. It should be enough to allocate MaxMetaspaceSize of compressed class
> space. Unfortunately, we don't know how much Metadata will be put inside
> the compressed class space vs the other metaspace memory areas (containing
> the non-class metadata). So, we would probably end up allowing ~ 2 x
> MaxMetaspaceSize of memory to be reserved if the program committed
> MaxMetaspaceSize of non-class metadata.
>
> We could reserve a contiguous memory area if user has set a low
> MaxMetaspaceSize, say below 1 GB, and use that area for both classes and
> non-class metadata. With that in place the MaxMetaspaceSize flag would
> limit both the reserved and committed memory. Unfortunately, this has a
> draw-back that mmap will destroy the reservation of the memory area if a
> sub-sequent "commit" request would fail. We would have to handle that
> somehow.
>
>    Or if it does use it, I'm assuming
>> it'll change protection on it and then possibly oom at that point (if
>> commit charge is over the limit).
>>
>> But anyway, back to the question at hand.  Is there a single flag that
>> would cap the amount of VA requested? It seems odd that MaxMetaspaceSize
>> by
>> itself doesn't change VA reservation, but adding CompressedClassSpaceSize
>> does.  Moreover, if I'm capping max size, why even reserve more VA in that
>> case?
>>
>
> MaxMetaspaceSize is designed to limit the amount of committed memory and
> not to limit the amount of reserved memory. At this moment, there are no
> flag to limit the amount of reserved memory by the metaspaces. If that
> feature is needed, it would have to be implemented.
>
> Thanks,
> StefanK
>
>
>> Thanks
>>
>> sent from my phone
>> On Apr 7, 2015 9:38 PM, "Srinivas Ramakrishna" <ysr1729 at gmail.com> wrote:
>>
>>  Hi Vitaly --
>>>
>>> Let me give an example: In a multi-threaded process, if we want some
>>> piece
>>> of anon memory to be placed at a specific spot in our (i.e. process's) VA
>>> address space, we can first "reserve" that address space (tit is in that
>>> sense purely local to the process, there are no physical pages backing
>>> it,
>>> no swap reservation, no OS accounting, nothing). We then can decide at
>>> leisure what to do with that address space, knowing that that address
>>> space
>>> in our process is reserved for us and "no one can take that away from
>>> us."
>>> This is what the JVM calls an address space reservation.
>>>
>>> What the JVM calls "commit" is when we reserve the virtual memory with
>>> the
>>> OS. Committed memory may or may not be backed by physical pages. (For
>>> example, on Linux wjere you allow overcommit.) It's only when you access
>>> that page will it be faulted in from the set of free pages in RAM (if
>>> there
>>> are any left and you aren't overcommitted).
>>>
>>> What is confusing you (and many others) is the use of the term
>>> "reserved".
>>> Here reserved is used in the sense of reserving the address space in the
>>> process. Nothing else. No OS book-keeping to reserve virtual *memory*,
>>> nothing.
>>>
>>> At least that was what it was a couple of years ago, but I haven't looked
>>> at the code in recent times (but given this discussion, I'll go and do so
>>> soon!).
>>> -- ramki
>>>
>>>
>>> On Tue, Apr 7, 2015 at 5:47 PM, Vitaly Davidovich <vitalyd at gmail.com>
>>> wrote:
>>>
>>>  Ramki,
>>>>
>>>> I'm not sure I fully understand your distinction between virtual address
>>>> space vs virtual memory.  If you disable swap and overcommit and you
>>>> mmap/brk from the kernel, it has to ensure that the reservation could be
>>>> backed up if it were committed; if you take away swap and don't allow
>>>> linux
>>>> to overpromise (i.e. overcommit), the malloc will fail.  See here for
>>>> what
>>>> I'm talking about: http://www.etalabs.net/overcommit.html
>>>>
>>>> The committed memory is memory that is now backed by physical ram (or
>>>> swap), it has either been touched or manually committed - it's not a
>>>> reservation (which has no physical backing, just kernel VM accounting
>>>> behind it).
>>>>
>>>> Did I misunderstand you?
>>>>
>>>> sent from my phone
>>>> Hi Vitaly --
>>>>
>>>> The allocated (but not necessarily touched/faulted) memory reported
>>>> above
>>>> (by the jvm) is the one corresponding to "committed". The "reservation"
>>>> reported above is only of the virtual *address space* within the
>>>> process,
>>>> not virtual *memory*  (i.e. no swap reservations or the like). I didn't
>>>> know that ulimits could prevent virtual address space (_not_ virtual
>>>> memory) reservation.
>>>>
>>>> I think your question is valid, in that looking at the VA reservation,
>>>> one feels that the JVM might eventually decide to commit beyond the max
>>>> space you asked for in your command-line spec. At the very least it is
>>>> confusing, and someone who knows why could explain the reason for the
>>>> larger than user-requested VA space (not VA memory yet in yr snippet)
>>>> reservation when it won't be used  -- if it won't be used, and here I am
>>>> giving the benefit of the doubt to the JVM in that it'll never try to
>>>> commit those pages despite having reserved that in its virtual address
>>>> space.
>>>>
>>>> -- ramki
>>>>
>>>> On Tue, Apr 7, 2015 at 5:01 PM, Vitaly Davidovich <vitalyd at gmail.com>
>>>> wrote:
>>>>
>>>>  By the way, forgot to mention that while doing some searching around
>>>>> this, saw this: https://issues.apache.org/jira/browse/HADOOP-11364.
>>>>> So
>>>>> anyone with multi-tenant servers that impose mem limits may hit this.
>>>>> But
>>>>> again, I'm trying to understand the flags involved and not saying this
>>>>> is a
>>>>> big problem.
>>>>>
>>>>> sent from my phone
>>>>> On Apr 7, 2015 7:55 PM, "Vitaly Davidovich" <vitalyd at gmail.com> wrote:
>>>>>
>>>>>  Hi Ramki,
>>>>>>
>>>>>> I agree in principle that a 1GB reservation on 64bit isn't an issue in
>>>>>> and of itself.  The problem comes up, however, if a program is run
>>>>>> with
>>>>>> ulimit on VA and someone sized their java app appropriately in, say,
>>>>>> java
>>>>>> 7.  With java 8, there can be regressions.
>>>>>>
>>>>>> Second, if you disable swap and overcommit on linux, larger virtual
>>>>>> mem
>>>>>> allocations may fail.
>>>>>>
>>>>>> Having said all that, I'm curious whether there's a way to restrict
>>>>>> this with 1 flag in java 8 and whether the linked documentation is
>>>>>> correct
>>>>>> or not.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> sent from my phone
>>>>>> On Apr 7, 2015 7:26 PM, "Srinivas Ramakrishna" <ysr1729 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>  Hi Vitaly --
>>>>>>>
>>>>>>> Because reserved but unused virtual address space is, for all
>>>>>>> practical purposes today, irrelevant for 64 bit processes,
>>>>>>> a question to ask is what you see when you do this for 32-bit
>>>>>>> processes (where of course the oop compression
>>>>>>> comments wouldn't of course apply, but the amount of VA space used
>>>>>>> would be much more relevant and parsimony
>>>>>>> -- or lack thereof -- in VA space reservation would visibly affect
>>>>>>> the
>>>>>>> bottom-line in a 32-bit VA space).
>>>>>>>
>>>>>>> -- ramki
>>>>>>>
>>>>>>> On Tue, Apr 7, 2015 at 12:27 PM, Vitaly Davidovich <
>>>>>>> vitalyd at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>  Hi guys,
>>>>>>>>
>>>>>>>> It seems that putting a cap on MaxMetaspaceSize has no impact on
>>>>>>>> amount of
>>>>>>>> memory reserved *unless* CompressedClassSpaceSize is also set.  For
>>>>>>>> example,
>>>>>>>>
>>>>>>>> java -Xmx64m -XX:MaxMetaspaceSize=16m Test
>>>>>>>>
>>>>>>>> yields the following:
>>>>>>>>
>>>>>>>>   Metaspace       used 2421K, capacity 4486K, committed 4864K,
>>>>>>>> reserved
>>>>>>>> 1056768K
>>>>>>>>    class space    used 261K, capacity 386K, committed 512K, reserved
>>>>>>>> 1048576K
>>>>>>>>
>>>>>>>> Adding -XX:CompressedClassSpaceSize=16m yields:
>>>>>>>>
>>>>>>>> Metaspace       used 2421K, capacity 4486K, committed 4864K,
>>>>>>>> reserved
>>>>>>>> 24576K
>>>>>>>>    class space    used 261K, capacity 386K, committed 512K, reserved
>>>>>>>> 16384K
>>>>>>>>
>>>>>>>> According to
>>>>>>>>
>>>>>>>> http://docs.oracle.com/javase/8/docs/technotes/guides/vm/
>>>>>>>> gctuning/considerations.html
>>>>>>>> :
>>>>>>>>
>>>>>>>>  If UseCompressedOops is turned on and UseCompressedClassesPointers
>>>>>>>>>
>>>>>>>> is
>>>>>>>>
>>>>>>>>> used, then two logically different areas of native memory are used
>>>>>>>>>
>>>>>>>> for
>>>>>>>>
>>>>>>>>> class metadata. UseCompressedClassPointers uses a 32-bit offset to
>>>>>>>>> represent the class pointer in a 64-bit process as does
>>>>>>>>>
>>>>>>>> UseCompressedOops for
>>>>>>>>
>>>>>>>>> Java object references. A region is allocated for these compressed
>>>>>>>>>
>>>>>>>> class
>>>>>>>>
>>>>>>>>> pointers (the 32-bit offsets). The size of the region can be set
>>>>>>>>>
>>>>>>>> with
>>>>>>>>
>>>>>>>>> CompressedClassSpaceSize and is 1 gigabyte (GB) by default. The
>>>>>>>>>
>>>>>>>> space for
>>>>>>>>
>>>>>>>>> the compressed class pointers is reserved as space allocated by
>>>>>>>>>
>>>>>>>> mmap at
>>>>>>>>
>>>>>>>>> initialization and committed as needed. The MaxMetaspaceSize
>>>>>>>>>
>>>>>>>> applies to
>>>>>>>>
>>>>>>>>> the sum of the committed compressed class space and the space for
>>>>>>>>>
>>>>>>>> the other
>>>>>>>>
>>>>>>>>> class metadata.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Unless I'm misunderstanding, the last sentence there seems to imply
>>>>>>>> that
>>>>>>>> MaxMetaspaceSize would put a cap on the compressed class space as
>>>>>>>> well.  Or
>>>>>>>> is the "committed" part of the last sentence the key? There's no way
>>>>>>>> to cap
>>>>>>>> the *reserved* amount with 1 flag? Basically, what's the java 8
>>>>>>>> equivalent
>>>>>>>> to MaxPermSize=X in java 7 and earlier?
>>>>>>>>
>>>>>>>> The reason this came up is because java 8 seems to have a much
>>>>>>>> higher
>>>>>>>> virtual mem reserve charge than java 7, even for an app with a
>>>>>>>> simple
>>>>>>>> main
>>>>>>>> that does nothing, and metaspace/compressed class space reservation
>>>>>>>> appears
>>>>>>>> to be the biggest contributor.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>
>