<div dir="ltr">Hi Mikael,<div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 11, 2016 at 11:26 AM, Mikael Gerdin <span dir="ltr"><<a href="mailto:mikael.gerdin@oracle.com" target="_blank">mikael.gerdin@oracle.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Thomas,<br>
<br>
I have some questions about how your suggested change affects the allocation policy.<br>
<br>
Currently, once a SpaceManager has reached the _small_chunk_limit it will only allocate medium chunks even if an allocation request could be satisfied with a smaller variant.<br></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">If metaspace is sufficiently fragmented by small chunks such that a medium chunk cannot be coalesced then we can still fail to satisfy an allocation request with large amounts of free memory, is this something that you have observed?<br></blockquote><div><br></div><div>I did not see this in my tests, but in theory this is still possible even with my patch. If free and in-use small chunks are tightly interspersed, this would of course still prevent merging them to medium chunks. I think to prevent this the chunk would have to be movable, so that they can be compacted. Or, as a quick fix, the size ratio between small and medium chunks could be made smaller, so that the chance that one stray in-use small chunk prevents its free neighbors from being coalesced is smaller.</div><div><br></div><div>In the scenarios we observed we saw mostly free small chunks resting in the freelist, because the class loaders which allocated them were all unloaded.<br></div><div><br></div><div>I try to understand how probable the scenario is you describe: in order to fill up metaspace with small chunks - so that allocation of new medium chunks fails - and to intermix free and in-use small chunks such that coalescation to a medium chunk is not possible but there are still enough free small "uncoalescable" chunks to matter (what you call "large amounts of free memory"), one would have to have a quite "unlucky" allocation pattern.</div><div><br></div><div>I think it is certainly possible, but less probable than the scenario this patch tries to address: that metaspace was allocated in many small chunks by class loaders which are now all dead and the small chunks are all free. </div><div><br></div><div>Kind Regards, Thomas</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks<span class="gmail-HOEnZb"><font color="#888888"><br>
/Mikael</font></span><span class="gmail-"><br>
<br>
On 2016-10-10 10:56, Thomas Stüfe wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">
Hi all,<br>
<br>
May I have please some feedback for this enhancement proposal?<br>
<br>
<a href="https://bugs.openjdk.java.net/browse/JDK-8166690" rel="noreferrer" target="_blank">https://bugs.openjdk.java.net/<wbr>browse/JDK-8166690</a><br>
<<a href="https://bugs.openjdk.java.net/browse/JDK-8166690" rel="noreferrer" target="_blank">https://bugs.openjdk.java.net<wbr>/browse/JDK-8166690</a>><br>
<br>
<br>
In one very short sentence it proposes a better allocation scheme for<br>
Metaspace Chunks in order to reduce fragmentation and metaspace waste.<br>
<br>
I also added a short presentation which describes the problem and how we<br>
solved it in our VM port.<br>
<br>
<a href="https://bugs.openjdk.java.net/secure/attachment/63894/Metaspace%20Coalescation%20in%20the%20SAP%20JVM.pdf" rel="noreferrer" target="_blank">https://bugs.openjdk.java.net/<wbr>secure/attachment/63894/Metasp<wbr>ace%20Coalescation%20in%20the%<wbr>20SAP%20JVM.pdf</a><br>
<br>
Thank you very much!<br>
<br>
Kind Regards, Thomas<br>
<br>
<br>
On Tue, Sep 27, 2016 at 10:45 AM, Thomas Stüfe <<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a><br></span><div><div class="gmail-h5">
<mailto:<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.co<wbr>m</a>>> wrote:<br>
<br>
Dear all,<br>
<br>
please take a look at this Enhancement Proposal for the metaspace<br>
allocator. I hope these are the right groups for this discussion.<br>
<br>
<a href="https://bugs.openjdk.java.net/browse/JDK-8166690" rel="noreferrer" target="_blank">https://bugs.openjdk.java.net/<wbr>browse/JDK-8166690</a><br>
<<a href="https://bugs.openjdk.java.net/browse/JDK-8166690" rel="noreferrer" target="_blank">https://bugs.openjdk.java.net<wbr>/browse/JDK-8166690</a>><br>
<br>
Background:<br>
<br>
We at SAP see at times at customer installations OOMs in Metaspace<br>
(usually, with compressed class pointers enabled, in Compressed<br>
Class Space). The VM attempts to allocate metaspace and fails,<br>
hitting the CompressedClassSpaceSize limit. Note that we usually set<br>
the limit lower than the default, typically at 256M.<br>
<br>
When analyzing, we observed that a large part of the metaspace is<br>
indeed free but "locked in" into metaspace chunks of the wrong size:<br>
often we would find a lot of free small chunks, but the allocation<br>
request was for medium chunks, and failed.<br>
<br>
The reason was that if at some point in time a lot of class loaders<br>
were alive, each with only a few small classes loaded. This would<br>
lead to the metaspace being swamped with lots of small chunks. This<br>
is because each SpaceManager first allocates small chunks, only<br>
after a certain amount of allocation requests switches to larger chunks.<br>
<br>
These small chunks are free and wait in the freelist, but cannot be<br>
reused for allocation requests which require larger chunks, even if<br>
they are physically adjacent in the virtual space.<br>
<br>
We (at SAP) added a patch which allows on-the-fly metaspace chunk<br>
merging - to merge multiple adjacent smaller chunk to form a larger<br>
chunk. This, in combination with the reverse direction - splitting a<br>
large chunk to get smaller chunks - partly negates the<br>
"chunks-are-locked-in-into-the<wbr>ir-size" limitation and provides for<br>
better reuse of metaspace chunks. It also provides better<br>
defragmentation as well.<br>
<br>
I discussed this fix off-list with Coleen Phillimore and Jon<br>
Masamitsu, and instead of just offering this as a fix, both<br>
recommended to open a JEP for this, because its scope would be<br>
beyond that of a simple fix.<br>
<br>
So here is my first JEP :) I hope it follows the right form. Please,<br>
if you have time, take a look and tell us what you think.<br>
<br>
Thank you, and Kind Regards,<br>
<br>
Thomas Stüfe<br>
<br>
<br>
<br>
<br>
</div></div></blockquote>
</blockquote></div><br></div></div>