RFR(S) 8189688: NMT: Report per-class load metadata information

Thomas Stüfe thomas.stuefe at gmail.com
Tue Oct 24 09:11:59 UTC 2017


Hi Andrew,

On Mon, Oct 23, 2017 at 5:29 PM, Andrew Dinn <adinn at redhat.com> wrote:

> On 23/10/17 16:03, Thomas Stüfe wrote:
> > I am confused about the sizes though . Are we talking about the chunk
> > sizes in metaspace.cpp? Because they are way smaller, with medium chunks
> > being 32/64K? Or are these large chunks a RedHat addition?
> Oops, apologies -- wrong units!
>
> Yes, this is indeed chunk sizes but I meant to say 1KB and 4KB chunks
> (MBs would be truly dreadful :-).


That makes more sense :)



> The relevant definitions in
> metaspace.cpp are:
>
> enum ChunkSizes {    // in words.
>   ClassSpecializedChunk = 128,
>   SpecializedChunk = 128,
>   ClassSmallChunk = 256,
>   SmallChunk = 512,
>   ClassMediumChunk = 4 * K,
>   MediumChunk = 8 * K
> };
>
> Anonymous classes (used for the anon classes used to back lambdas)
> initially allocate a ClassSpecializedChunk and a SpecializedChunk i.e. 2
> x 128 * 8 bytes = 2 x 1KB. If you have something that goes beyond the
> most basic lambda then you may well also allocate an extra SmallChunk =
> 512 * 8 bytes = 4KB. We experimented (with both real and synthetic
> examples) and found a high proportion of cases where 6KB chunks was
> being allocated but around 50% was left unused as free tail ends of
> these 2/3 chunks.
>
> regards,
>
>
>
Last year I analyzed some OOM-in-classspace situations. I saw this problem,
but was
disregarding it as something which would have diminishing effect with
increased number
of loaded classes. But now I see that this is indeed a real world problem.

(The actual problem in my case turned out to be the class space being
flooded with small chunks,
which were free because its creating class loaders have been unloaded, but
due to the unability
of the metaspace coding to reuse chunks in a different size were not
reused. We added a patch
in our fork to split-and-merge free metaspace chunks on demand to deal with
this problem.)

My first impulse was to say "Well, lets make the _small_chunk_limit
configurable via command line",
but I think that is a short sighted stop gap. A better solution may be to
make _small_chunk_limit  a
per-classloader-setting, so that class loaders which we know will only be
used for some anonymous
classes can be started with a high _small_chunk_limit  to never let them
allocate medium chunks.

I wonder if another solution could be some sort of stealing mechanism:
Periodically walk all class loaders,
and, if they have half-eaten medium chunks laying around they did not touch
for a while, split that chunk into
n small chunks and return n-1 chunks to the free list. This only works
though if the "used" portion of the
medium chunk does not surpass the size of one small chunk, because the
portion in use cannot be split
into multiple chunks. Considering that the size ratio small:medium is 1:16,
this may not be that useful. As
you say, usually the medium chunks are used up about half way.

Kind Regards, Thomas


> Andrew Dinn
> -----------
> Senior Principal Software Engineer
> Red Hat UK Ltd
> Registered in England and Wales under Company Registration No. 03798903
> Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
>


More information about the hotspot-runtime-dev mailing list