jdk 8u40 - MaxMetaspaceSize and CompressedClassSpaceSize

Vitaly Davidovich vitalyd at gmail.com
Wed Apr 8 15:20:14 UTC 2015 

Ok, thanks Stefan.  If it's not too difficult and doesn't make other things
worse, I think it's preferable to change this to not over reserve.  If
nothing else, it's potentially confusing to see committed space capped but
still have a large reservation there.  Also, things like ulimit that
restrict VA amount may barf in those cases unnecessarily.

On Wed, Apr 8, 2015 at 10:29 AM, Stefan Karlsson <stefan.karlsson at oracle.com
> wrote:

>  Hi Vitaly,
>
> On 2015-04-08 15:06, Vitaly Davidovich wrote:
>
> Hi Stefan,
>
> Thanks for clarifying some of this.  The one thing I'm still unclear about
> is why more VA is reserved when a max committed amount is set to a lower
> value.  I'm assuming that if I set MaxMetaspaceSize to, say, 32MB then
> trying to use/commit more than that will result in an OOME.
>
>
> That's correct.
>
>  Yet the VA reservation stays at 1GB.
>
>
> Yes, that's how the compressed class space was chosen to be implemented to
> make sure that the Klasses end up at an address that we can compress down
> to 32-bits. This could be changed to reserve less memory when
> MaxMetaspaceSize is set to a lower value.
>
>  Or are you saying that 32MB only restricts the non-class metadata and
> thus you need to leave additional room for class metadata? Does
> MaxMetaspaceSize actually cap both class and non-class metadata? The doc I
> linked to earlier suggests it caps both, but if I'm reading your email
> right, you seem to be saying that even if MaxMetaspaceSize is set you still
> need additional room for class metadata.
>
>
> MaxMetspaceSize caps the combined _committed_ memory, so the process will
> not commit more space than what you've specified on the command line.
>
> What I'm trying to say is that having the current implementation of
> compressed class space forces us to pre-reserve at least 32MB even if we
> are not going to use/commit all that memory. Since we don't know the
> distribution between Klasses and non-class metadata, we also have to allow
> the non-class metadata spaces to reserve up to 32MB as well. So, in the
> extreme case we will reserve 64MB but we'll not commit more than 32MB.
>
> However, as you've noted we pre-reserve 1GB, and not 32 MB, for the
> compressed class space. That can be fixed by either running with
> -XX:CompressedClassSpaceSize=32m, or by a patch to change the
> implementation to only reserve MIN(CompressedClassSpaceSize,
> MaxMetaspaceSize) for the compressed class space.
>
>
> Thanks,
> StefanK
>
>  Thanks
>
> sent from my phone
> On Apr 8, 2015 5:23 AM, "Stefan Karlsson" <stefan.karlsson at oracle.com>
> wrote:
>
>> Hi Vitaly,
>>
>> On 2015-04-08 04:26, Vitaly Davidovich wrote:
>>
>>> Thanks Ramki, that does clear it up a bit.  So you're saying the JVM
>>> notion
>>> of reserved is basically (on linux) mmap()'ing anon memory with PROT_NONE
>>> equivalent - i.e. no commit charge.  Is that right?
>>>
>>
>> That's sounds right to me.
>>
>> The code path to the mmap:
>> os::reserve_memory
>>   os::pd_reserve_memory
>>     anon_mmap
>>       mmap( ... PROT_NONE ... , ... MAP_NORESERVE ... )
>>
>> I did an experiment with some time ago that can be found at:
>> http://mail.openjdk.java.net/pipermail/hotspot-dev/2013-July/010226.html
>>
>> Independent of the over-commit settings in the Linux kernel, I could
>> still "reserve" 2TB of memory on my 8GB machine, if PROT_NONE was used.
>>
>>
>>> Ok, so the issue with ulimit is probably that it actually restricts
>>> address
>>> space, and not virtual memory.
>>>
>>> As for the reservation in Hotspot, why is it reserving that address space
>>> if it has no intention of using it?
>>>
>>
>> I agree that we shouldn't have to reserve 1 GB of virtual address space
>> for the compressed class space when MaxMetaspaceSize is set to  a lower
>> value. It should be enough to allocate MaxMetaspaceSize of compressed class
>> space. Unfortunately, we don't know how much Metadata will be put inside
>> the compressed class space vs the other metaspace memory areas (containing
>> the non-class metadata). So, we would probably end up allowing ~ 2 x
>> MaxMetaspaceSize of memory to be reserved if the program committed
>> MaxMetaspaceSize of non-class metadata.
>>
>> We could reserve a contiguous memory area if user has set a low
>> MaxMetaspaceSize, say below 1 GB, and use that area for both classes and
>> non-class metadata. With that in place the MaxMetaspaceSize flag would
>> limit both the reserved and committed memory. Unfortunately, this has a
>> draw-back that mmap will destroy the reservation of the memory area if a
>> sub-sequent "commit" request would fail. We would have to handle that
>> somehow.
>>
>>    Or if it does use it, I'm assuming
>>> it'll change protection on it and then possibly oom at that point (if
>>> commit charge is over the limit).
>>>
>>> But anyway, back to the question at hand.  Is there a single flag that
>>> would cap the amount of VA requested? It seems odd that MaxMetaspaceSize
>>> by
>>> itself doesn't change VA reservation, but adding CompressedClassSpaceSize
>>> does.  Moreover, if I'm capping max size, why even reserve more VA in
>>> that
>>> case?
>>>
>>
>> MaxMetaspaceSize is designed to limit the amount of committed memory and
>> not to limit the amount of reserved memory. At this moment, there are no
>> flag to limit the amount of reserved memory by the metaspaces. If that
>> feature is needed, it would have to be implemented.
>>
>> Thanks,
>> StefanK
>>
>>
>>> Thanks
>>>
>>> sent from my phone
>>> On Apr 7, 2015 9:38 PM, "Srinivas Ramakrishna" <ysr1729 at gmail.com>
>>> wrote:
>>>
>>>  Hi Vitaly --
>>>>
>>>> Let me give an example: In a multi-threaded process, if we want some
>>>> piece
>>>> of anon memory to be placed at a specific spot in our (i.e. process's)
>>>> VA
>>>> address space, we can first "reserve" that address space (tit is in that
>>>> sense purely local to the process, there are no physical pages backing
>>>> it,
>>>> no swap reservation, no OS accounting, nothing). We then can decide at
>>>> leisure what to do with that address space, knowing that that address
>>>> space
>>>> in our process is reserved for us and "no one can take that away from
>>>> us."
>>>> This is what the JVM calls an address space reservation.
>>>>
>>>> What the JVM calls "commit" is when we reserve the virtual memory with
>>>> the
>>>> OS. Committed memory may or may not be backed by physical pages. (For
>>>> example, on Linux wjere you allow overcommit.) It's only when you access
>>>> that page will it be faulted in from the set of free pages in RAM (if
>>>> there
>>>> are any left and you aren't overcommitted).
>>>>
>>>> What is confusing you (and many others) is the use of the term
>>>> "reserved".
>>>> Here reserved is used in the sense of reserving the address space in the
>>>> process. Nothing else. No OS book-keeping to reserve virtual *memory*,
>>>> nothing.
>>>>
>>>> At least that was what it was a couple of years ago, but I haven't
>>>> looked
>>>> at the code in recent times (but given this discussion, I'll go and do
>>>> so
>>>> soon!).
>>>> -- ramki
>>>>
>>>>
>>>> On Tue, Apr 7, 2015 at 5:47 PM, Vitaly Davidovich <vitalyd at gmail.com>
>>>> wrote:
>>>>
>>>>  Ramki,
>>>>>
>>>>> I'm not sure I fully understand your distinction between virtual
>>>>> address
>>>>> space vs virtual memory.  If you disable swap and overcommit and you
>>>>> mmap/brk from the kernel, it has to ensure that the reservation could
>>>>> be
>>>>> backed up if it were committed; if you take away swap and don't allow
>>>>> linux
>>>>> to overpromise (i.e. overcommit), the malloc will fail.  See here for
>>>>> what
>>>>> I'm talking about: http://www.etalabs.net/overcommit.html
>>>>>
>>>>> The committed memory is memory that is now backed by physical ram (or
>>>>> swap), it has either been touched or manually committed - it's not a
>>>>> reservation (which has no physical backing, just kernel VM accounting
>>>>> behind it).
>>>>>
>>>>> Did I misunderstand you?
>>>>>
>>>>> sent from my phone
>>>>> Hi Vitaly --
>>>>>
>>>>> The allocated (but not necessarily touched/faulted) memory reported
>>>>> above
>>>>> (by the jvm) is the one corresponding to "committed". The "reservation"
>>>>> reported above is only of the virtual *address space* within the
>>>>> process,
>>>>> not virtual *memory*  (i.e. no swap reservations or the like). I didn't
>>>>> know that ulimits could prevent virtual address space (_not_ virtual
>>>>> memory) reservation.
>>>>>
>>>>> I think your question is valid, in that looking at the VA reservation,
>>>>> one feels that the JVM might eventually decide to commit beyond the max
>>>>> space you asked for in your command-line spec. At the very least it is
>>>>> confusing, and someone who knows why could explain the reason for the
>>>>> larger than user-requested VA space (not VA memory yet in yr snippet)
>>>>> reservation when it won't be used  -- if it won't be used, and here I
>>>>> am
>>>>> giving the benefit of the doubt to the JVM in that it'll never try to
>>>>> commit those pages despite having reserved that in its virtual address
>>>>> space.
>>>>>
>>>>> -- ramki
>>>>>
>>>>> On Tue, Apr 7, 2015 at 5:01 PM, Vitaly Davidovich <vitalyd at gmail.com>
>>>>> wrote:
>>>>>
>>>>>  By the way, forgot to mention that while doing some searching around
>>>>>> this, saw this: https://issues.apache.org/jira/browse/HADOOP-11364.
>>>>>> So
>>>>>> anyone with multi-tenant servers that impose mem limits may hit
>>>>>> this.  But
>>>>>> again, I'm trying to understand the flags involved and not saying
>>>>>> this is a
>>>>>> big problem.
>>>>>>
>>>>>> sent from my phone
>>>>>> On Apr 7, 2015 7:55 PM, "Vitaly Davidovich" <vitalyd at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>  Hi Ramki,
>>>>>>>
>>>>>>> I agree in principle that a 1GB reservation on 64bit isn't an issue
>>>>>>> in
>>>>>>> and of itself.  The problem comes up, however, if a program is run
>>>>>>> with
>>>>>>> ulimit on VA and someone sized their java app appropriately in, say,
>>>>>>> java
>>>>>>> 7.  With java 8, there can be regressions.
>>>>>>>
>>>>>>> Second, if you disable swap and overcommit on linux, larger virtual
>>>>>>> mem
>>>>>>> allocations may fail.
>>>>>>>
>>>>>>> Having said all that, I'm curious whether there's a way to restrict
>>>>>>> this with 1 flag in java 8 and whether the linked documentation is
>>>>>>> correct
>>>>>>> or not.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> sent from my phone
>>>>>>> On Apr 7, 2015 7:26 PM, "Srinivas Ramakrishna" <ysr1729 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>  Hi Vitaly --
>>>>>>>>
>>>>>>>> Because reserved but unused virtual address space is, for all
>>>>>>>> practical purposes today, irrelevant for 64 bit processes,
>>>>>>>> a question to ask is what you see when you do this for 32-bit
>>>>>>>> processes (where of course the oop compression
>>>>>>>> comments wouldn't of course apply, but the amount of VA space used
>>>>>>>> would be much more relevant and parsimony
>>>>>>>> -- or lack thereof -- in VA space reservation would visibly affect
>>>>>>>> the
>>>>>>>> bottom-line in a 32-bit VA space).
>>>>>>>>
>>>>>>>> -- ramki
>>>>>>>>
>>>>>>>> On Tue, Apr 7, 2015 at 12:27 PM, Vitaly Davidovich <
>>>>>>>> vitalyd at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>  Hi guys,
>>>>>>>>>
>>>>>>>>> It seems that putting a cap on MaxMetaspaceSize has no impact on
>>>>>>>>> amount of
>>>>>>>>> memory reserved *unless* CompressedClassSpaceSize is also set.  For
>>>>>>>>> example,
>>>>>>>>>
>>>>>>>>> java -Xmx64m -XX:MaxMetaspaceSize=16m Test
>>>>>>>>>
>>>>>>>>> yields the following:
>>>>>>>>>
>>>>>>>>>   Metaspace       used 2421K, capacity 4486K, committed 4864K,
>>>>>>>>> reserved
>>>>>>>>> 1056768K
>>>>>>>>>    class space    used 261K, capacity 386K, committed 512K,
>>>>>>>>> reserved
>>>>>>>>> 1048576K
>>>>>>>>>
>>>>>>>>> Adding -XX:CompressedClassSpaceSize=16m yields:
>>>>>>>>>
>>>>>>>>> Metaspace       used 2421K, capacity 4486K, committed 4864K,
>>>>>>>>> reserved
>>>>>>>>> 24576K
>>>>>>>>>    class space    used 261K, capacity 386K, committed 512K,
>>>>>>>>> reserved
>>>>>>>>> 16384K
>>>>>>>>>
>>>>>>>>> According to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.html
>>>>>>>>> :
>>>>>>>>>
>>>>>>>>>  If UseCompressedOops is turned on and UseCompressedClassesPointers
>>>>>>>>>>
>>>>>>>>> is
>>>>>>>>>
>>>>>>>>>> used, then two logically different areas of native memory are used
>>>>>>>>>>
>>>>>>>>> for
>>>>>>>>>
>>>>>>>>>> class metadata. UseCompressedClassPointers uses a 32-bit offset to
>>>>>>>>>> represent the class pointer in a 64-bit process as does
>>>>>>>>>>
>>>>>>>>> UseCompressedOops for
>>>>>>>>>
>>>>>>>>>> Java object references. A region is allocated for these compressed
>>>>>>>>>>
>>>>>>>>> class
>>>>>>>>>
>>>>>>>>>> pointers (the 32-bit offsets). The size of the region can be set
>>>>>>>>>>
>>>>>>>>> with
>>>>>>>>>
>>>>>>>>>> CompressedClassSpaceSize and is 1 gigabyte (GB) by default. The
>>>>>>>>>>
>>>>>>>>> space for
>>>>>>>>>
>>>>>>>>>> the compressed class pointers is reserved as space allocated by
>>>>>>>>>>
>>>>>>>>> mmap at
>>>>>>>>>
>>>>>>>>>> initialization and committed as needed. The MaxMetaspaceSize
>>>>>>>>>>
>>>>>>>>> applies to
>>>>>>>>>
>>>>>>>>>> the sum of the committed compressed class space and the space for
>>>>>>>>>>
>>>>>>>>> the other
>>>>>>>>>
>>>>>>>>>> class metadata.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Unless I'm misunderstanding, the last sentence there seems to imply
>>>>>>>>> that
>>>>>>>>> MaxMetaspaceSize would put a cap on the compressed class space as
>>>>>>>>> well.  Or
>>>>>>>>> is the "committed" part of the last sentence the key? There's no
>>>>>>>>> way
>>>>>>>>> to cap
>>>>>>>>> the *reserved* amount with 1 flag? Basically, what's the java 8
>>>>>>>>> equivalent
>>>>>>>>> to MaxPermSize=X in java 7 and earlier?
>>>>>>>>>
>>>>>>>>> The reason this came up is because java 8 seems to have a much
>>>>>>>>> higher
>>>>>>>>> virtual mem reserve charge than java 7, even for an app with a
>>>>>>>>> simple
>>>>>>>>> main
>>>>>>>>> that does nothing, and metaspace/compressed class space reservation
>>>>>>>>> appears
>>>>>>>>> to be the biggest contributor.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>
>

More information about the hotspot-runtime-dev mailing list