<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<br>
<div class="moz-cite-prefix">On 10/25/16 3:39 AM, Thomas Stüfe
wrote:<br>
</div>
<blockquote
cite="mid:CAA-vtUxQP0q6DGVCC=KhBKOnW04XXfTAq=3r-bMYOvmS+3bc7w@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Coleen,
<div><br>
</div>
<div>thank you for feedback and encouragement :) See further
comments inline.</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Oct 24, 2016 at 6:32 PM,
Coleen Phillimore <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:coleen.phillimore@oracle.com"
target="_blank">coleen.phillimore@oracle.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi Thomas,<br>
<br>
I agree with Erik. If this works well for you, then it
should just be implemented without an option. If done
early in JDK10, it'll get a lot of good testing.<br>
</blockquote>
<div><br>
</div>
<div>Ok. Was hoping to get this into JDK9, but if I am
thinking clearly about this, I see that maybe the risk is
too large. So lets implement this in 10 and if it is
stable and works well, it can be backported to jdk9, yes?</div>
</div>
</div>
</div>
</blockquote>
<br>
That seems a better plan. Even though JDK9 slipped for jigsaw, we
are keeping to the original FC date for everything else, so that
it'll be very stable when it ships.<br>
<br>
JDK10 should open soon.<br>
<blockquote
cite="mid:CAA-vtUxQP0q6DGVCC=KhBKOnW04XXfTAq=3r-bMYOvmS+3bc7w@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
This looks like a very good improvement. We had
discussed coalescing blocks and other improvements like
this early on, but wanted to wait to see problems in the
field to motivate the changes. We've seen these sorts of
problems now too.<br>
<br>
One of the things we've considered is that we wanted to
use the operating system's version of malloc for the
chunks, so that it can split and coalesce chunks for us.
The Solaris malloc is improved recently. But I don't
think it's time to make change to use malloc'ed chunks yet
because we have to consider all of the operating systems
that we and the OpenJDK community supports.<br>
</blockquote>
<div><br>
</div>
<div>I do not see how you could get CompressedClassPointers
to work with native malloc? You would have to be sure that
the pointers returned by malloc are within the numerical
range for the 32bit class pointers. I thought that was the
reason for using a continuous address range when
allocating compressed class space. <br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Oh, yes, you are right. I thought we saw the too many small chunks
problem only in the data metaspace, but this would help with class
metaspace as well.<br>
<br>
Thanks,<br>
Coleen<br>
<br>
<blockquote
cite="mid:CAA-vtUxQP0q6DGVCC=KhBKOnW04XXfTAq=3r-bMYOvmS+3bc7w@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
So, yes, I think the JEP looks good and your slides are
absolutely beautiful. Everyone should see these slides.<br>
<br>
</blockquote>
<div><br>
</div>
<div>:) Thanks! </div>
<div><br>
</div>
<div>So I will provide a prototype, for now based on jdk9,
and we will see how we go from there. </div>
<div><br>
</div>
<div>Thanks, and Kind Regards, </div>
<div><br>
</div>
<div>Thomas</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks,<br>
Coleen
<div class="HOEnZb">
<div class="h5"><br>
<br>
<br>
On 10/24/16 10:01 AM, Erik Helin wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
On 2016-10-13, Thomas Stüfe wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Erik,<br>
<br>
On Thu, Oct 13, 2016 at 2:15 PM, Erik Helin <<a
moz-do-not-send="true"
href="mailto:erik.helin@oracle.com"
target="_blank">erik.helin@oracle.com</a>>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
Hi Thomas,<br>
<br>
thanks for submitting the JEP and proposing this
feature!<br>
<br>
On 2016-10-10, Thomas Stüfe wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
Hi all,<br>
<br>
May I have please some feedback for this
enhancement proposal?<br>
<br>
<a moz-do-not-send="true"
href="https://bugs.openjdk.java.net/browse/JDK-8166690"
rel="noreferrer" target="_blank">https://bugs.openjdk.java.net/<wbr>browse/JDK-8166690</a><br>
<br>
<br>
In one very short sentence it proposes a
better allocation scheme for<br>
Metaspace Chunks in order to reduce
fragmentation and metaspace waste.<br>
<br>
I also added a short presentation which
describes the problem and how we<br>
solved it in our VM port.<br>
<br>
<a moz-do-not-send="true"
href="https://bugs.openjdk.java.net/secure/attachment/63894/"
rel="noreferrer" target="_blank">https://bugs.openjdk.java.net/<wbr>secure/attachment/63894/</a><br>
</blockquote>
Metaspace%20Coalescation%20in%<wbr>20the%20SAP%20JVM.pdf<br>
<br>
Do we really need the flag
-XX:+CoalesceMetaspace? Having two differnent<br>
ways to handle the chunk free lists in Metaspace
is unfortunate, it<br>
might introduce hard to detect bugs and will
also require much more<br>
testing (runnings lots of tests with the flag
both on and off).<br>
<br>
</blockquote>
You are right. If the new allocator works well,
there is no reason to keep<br>
the old allocator around.<br>
<br>
We wanted for a temporary time to be able to
switch between both old and<br>
new allocator. Just to have a fallback if problems
occur. But if it works,<br>
it makes sense to only have one allocator, and the
"CoalesceMetaspace" flag<br>
can be removed, and also the code can be made a
lot simpler because we do<br>
not need both code paths.<br>
</blockquote>
Yeah, I would strongly prefer to not introduce a new
flag for this. Have<br>
you thought about testing? Do you intend to write
new tests to stress<br>
the coalescing?<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
Do you think your proposed solution has low
enough overhead (in terms<br>
of CPU and memory) to be on "by default"?<br>
<br>
</blockquote>
We decided to switch it on by default in our VM.<br>
<br>
Memory overhead can be almost exactly calculated.
Bitmasks take 2 bits per<br>
specialized-chunk-sized-area. That means, for
specialized-chunk-size = 1k<br>
(128 meta words): metaspace size / 8192. So, for
1G of metaspace we pay<br>
132KB overhead for the bitmasks, or roughly 0.1%.<br>
<br>
There is some CPU overhead, but in my tests I
could not measure anything<br>
above noise level.<br>
</blockquote>
Those numbers seems low enough to me in order to not
warrant a new flag.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
Thanks,<br>
Erik<br>
<br>
<br>
</blockquote>
Btw, I understand that it is difficult to estimate
this proposal without a<br>
prototype to play around. As I already mentioned,
the patch right now only<br>
exists in our code base and not yet in the
OpenJDK. If you guys are<br>
seriously interested in this JEP, I will invest
the time to port the patch<br>
to the OpenJDK, so that you can check it out for
yourself.<br>
</blockquote>
Yes, we are seriously interested :) I think the
proposal sounds good. I guess<br>
the devil will be in the details, so I (we) would
really appreciate if<br>
you want to port your internal patch to OpenJDK.<br>
<br>
Thanks,<br>
Erik<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Kind Regards, Thomas<br>
<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
Thank you very much!<br>
<br>
Kind Regards, Thomas<br>
<br>
<br>
On Tue, Sep 27, 2016 at 10:45 AM, Thomas Stüfe
<<a moz-do-not-send="true"
href="mailto:thomas.stuefe@gmail.com"
target="_blank">thomas.stuefe@gmail.com</a>><br>
wrote:<br>
<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
Dear all,<br>
<br>
please take a look at this Enhancement
Proposal for the metaspace<br>
allocator. I hope these are the right groups
for this discussion.<br>
<br>
<a moz-do-not-send="true"
href="https://bugs.openjdk.java.net/browse/JDK-8166690"
rel="noreferrer" target="_blank">https://bugs.openjdk.java.net/<wbr>browse/JDK-8166690</a><br>
<br>
Background:<br>
<br>
We at SAP see at times at customer
installations OOMs in Metaspace<br>
(usually, with compressed class pointers
enabled, in Compressed Class<br>
Space). The VM attempts to allocate
metaspace and fails, hitting the<br>
CompressedClassSpaceSize limit. Note that we
usually set the limit<br>
</blockquote>
</blockquote>
lower<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
than the default, typically at 256M.<br>
<br>
When analyzing, we observed that a large
part of the metaspace is<br>
</blockquote>
</blockquote>
indeed<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
free but "locked in" into metaspace chunks
of the wrong size: often we<br>
would find a lot of free small chunks, but
the allocation request was<br>
</blockquote>
</blockquote>
for<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
medium chunks, and failed.<br>
<br>
The reason was that if at some point in time
a lot of class loaders<br>
</blockquote>
</blockquote>
were<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
alive, each with only a few small classes
loaded. This would lead to<br>
</blockquote>
</blockquote>
the<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
metaspace being swamped with lots of small
chunks. This is because each<br>
SpaceManager first allocates small chunks,
only after a certain amount<br>
</blockquote>
</blockquote>
of<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
allocation requests switches to larger
chunks.<br>
<br>
These small chunks are free and wait in the
freelist, but cannot be<br>
</blockquote>
</blockquote>
reused<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
for allocation requests which require larger
chunks, even if they are<br>
physically adjacent in the virtual space.<br>
<br>
We (at SAP) added a patch which allows
on-the-fly metaspace chunk<br>
</blockquote>
</blockquote>
merging<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
- to merge multiple adjacent smaller chunk
to form a larger chunk.<br>
</blockquote>
</blockquote>
This, in<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
combination with the reverse direction -
splitting a large chunk to get<br>
smaller chunks - partly negates the
"chunks-are-locked-in-into-<br>
</blockquote>
</blockquote>
their-size"<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
limitation and provides for better reuse of
metaspace chunks. It also<br>
provides better defragmentation as well.<br>
<br>
I discussed this fix off-list with Coleen
Phillimore and Jon Masamitsu,<br>
and instead of just offering this as a fix,
both recommended to open a<br>
</blockquote>
</blockquote>
JEP<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
for this, because its scope would be beyond
that of a simple fix.<br>
<br>
So here is my first JEP :) I hope it follows
the right form. Please, if<br>
you have time, take a look and tell us what
you think.<br>
<br>
Thank you, and Kind Regards,<br>
<br>
Thomas Stüfe<br>
<br>
<br>
<br>
<br>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>