<div dir="ltr"><div>Yes, that's the space for the old bytecode sitting around waiting to be re-used:</div><div><br></div><div>`
Deallocated from chunks in use: 127.36 MB (>99%) (11785 blocks) `</div><div><br></div><div>It would be nice if you could open a JBS issue and attach the program and a log produced with "-Xlog:metaspace=trace"; you may have to zip the thing if it gets too large.<br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 24, 2023 at 9:06 AM Jean-Philippe Bempel <<a href="mailto:jean-philippe.bempel@datadoghq.com">jean-philippe.bempel@datadoghq.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, May 24, 2023 at 8:21 AM Thomas Stüfe <<a href="mailto:thomas.stuefe@gmail.com" target="_blank">thomas.stuefe@gmail.com</a>> wrote:<br>
><br>
><br>
> My first thought was that this is not a leak but a case of class redefinition, where the bytecode is rewritten. The space for the old bytecode gets salvaged and would be re-used for metaspace allocations from the same loader, save that said re-use never happens for some reason. The output of "jcmd VM.metaspace" at the end of the program run would be helpful too.<br>
><br>
Here the output of VM.metaspace:<br>
<br>
$ jcmd 6940 VM.metaspace<br>
6940:<br>
<br>
Total Usage - 6 loaders, 809 classes (776 shared):<br>
Non-Class: 2605 chunks, 131.55 MB capacity, 127.87 MB ( 97%)<br>
committed, 127.83 MB ( 97%) used, 37.94 KB ( <1%) free, 8<br>
bytes ( <1%) waste , deallocated: 11783 blocks with 127.36 MB<br>
Class: 8 chunks, 266.00 KB capacity, 74.00 KB ( 28%)<br>
committed, 17.48 KB ( 7%) used, 56.52 KB ( 21%) free, 0<br>
bytes ( 0%) waste , deallocated: 2 blocks with 512 bytes<br>
Both: 2613 chunks, 131.81 MB capacity, 127.94 MB ( 97%)<br>
committed, 127.85 MB ( 97%) used, 94.45 KB ( <1%) free, 8<br>
bytes ( <1%) waste , deallocated: 11785 blocks with 127.36 MB<br>
<br>
<br>
Virtual space:<br>
Non-class space: 192.00 MB reserved, 127.88 MB ( 67%)<br>
committed, 3 nodes.<br>
Class space: 104.00 MB reserved, 128.00 KB ( <1%)<br>
committed, 1 nodes.<br>
Both: 296.00 MB reserved, 128.00 MB ( 43%) committed.<br>
<br>
<br>
Chunk freelists:<br>
Non-Class:<br>
<br>
4m: (none)<br>
2m: (none)<br>
1m: (none)<br>
512k: (none)<br>
256k: 2, capacity=512.00 KB, committed=0 bytes ( 0%)<br>
128k: 2, capacity=256.00 KB, committed=0 bytes ( 0%)<br>
64k: 2, capacity=128.00 KB, committed=0 bytes ( 0%)<br>
32k: (none)<br>
16k: (none)<br>
8k: (none)<br>
4k: 2, capacity=8.00 KB, committed=8.00 KB (100%)<br>
2k: 6, capacity=12.00 KB, committed=12.00 KB (100%)<br>
1k: (none)<br>
Total word size: 916.00 KB, committed: 20.00 KB ( 2%)<br>
<br>
Class:<br>
<br>
4m: (none)<br>
2m: 1, capacity=2.00 MB, committed=0 bytes ( 0%)<br>
1m: 1, capacity=1.00 MB, committed=0 bytes ( 0%)<br>
512k: 1, capacity=512.00 KB, committed=0 bytes ( 0%)<br>
256k: (none)<br>
128k: 1, capacity=128.00 KB, committed=0 bytes ( 0%)<br>
64k: 1, capacity=64.00 KB, committed=0 bytes ( 0%)<br>
32k: 1, capacity=32.00 KB, committed=0 bytes ( 0%)<br>
16k: 1, capacity=16.00 KB, committed=0 bytes ( 0%)<br>
8k: (none)<br>
4k: (none)<br>
2k: 1, capacity=2.00 KB, committed=2.00 KB (100%)<br>
1k: 3, capacity=3.00 KB, committed=3.00 KB (100%)<br>
Total word size: 3.74 MB, committed: 5.00 KB ( <1%)<br>
<br>
Both:<br>
<br>
4m: (none)<br>
2m: 1, capacity=2.00 MB, committed=0 bytes ( 0%)<br>
1m: 1, capacity=1.00 MB, committed=0 bytes ( 0%)<br>
512k: 1, capacity=512.00 KB, committed=0 bytes ( 0%)<br>
256k: 2, capacity=512.00 KB, committed=0 bytes ( 0%)<br>
128k: 3, capacity=384.00 KB, committed=0 bytes ( 0%)<br>
64k: 3, capacity=192.00 KB, committed=0 bytes ( 0%)<br>
32k: 1, capacity=32.00 KB, committed=0 bytes ( 0%)<br>
16k: 1, capacity=16.00 KB, committed=0 bytes ( 0%)<br>
8k: (none)<br>
4k: 2, capacity=8.00 KB, committed=8.00 KB (100%)<br>
2k: 7, capacity=14.00 KB, committed=14.00 KB (100%)<br>
1k: 3, capacity=3.00 KB, committed=3.00 KB (100%)<br>
Total word size: 4.63 MB, committed: 25.00 KB ( <1%)<br>
<br>
<br>
<br>
Waste (unused committed space):(percentages refer to total committed<br>
size 128.00 MB):<br>
Waste in chunks in use: 8 bytes ( <1%)<br>
Free in chunks in use: 94.45 KB ( <1%)<br>
In free chunks: 25.00 KB ( <1%)<br>
Deallocated from chunks in use: 127.36 MB (>99%) (11785 blocks)<br>
-total-: 127.48 MB (>99%)<br>
<br>
chunk header pool: 2632 items, 189.33 KB.<br>
<br>
Internal statistics:<br>
<br>
num_allocs_failed_limit: 23.<br>
num_arena_births: 22.<br>
num_arena_deaths: 10.<br>
num_vsnodes_births: 4.<br>
num_vsnodes_deaths: 0.<br>
num_space_committed: 2048.<br>
num_space_uncommitted: 0.<br>
num_chunks_returned_to_freelist: 33.<br>
num_chunks_taken_from_freelist: 2624.<br>
num_chunk_merges: 11.<br>
num_chunk_splits: 1456.<br>
num_chunks_enlarged: 276.<br>
num_inconsistent_stats: 0.<br>
<br>
<br>
Settings:<br>
MaxMetaspaceSize: 128.00 MB<br>
CompressedClassSpaceSize: 104.00 MB<br>
Initial GC threshold: 21.00 MB<br>
Current GC threshold: 128.00 MB<br>
CDS: on<br>
MetaspaceReclaimPolicy: balanced<br>
- commit_granule_bytes: 65536.<br>
- commit_granule_words: 8192.<br>
- virtual_space_node_default_size: 8388608.<br>
- enlarge_chunks_in_place: 1.<br>
- new_chunks_are_fully_committed: 0.<br>
- uncommit_free_chunks: 1.<br>
- use_allocation_guard: 0.<br>
<br>
Also I have run it with -XX:+CrashOnOutOfMemoryError to have stacktrace:<br>
Stack: [0x00007f8d9c400000,0x00007f8d9c500000],<br>
sp=0x00007f8d9c4fdb80, free space=1014k<br>
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)<br>
V [libjvm.so+0xf9f4e7] VMError::report_and_die(int, char const*,<br>
char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*,<br>
char const*, int, unsigned long)+0x197 (debug.cpp:368)<br>
V [libjvm.so+0x672eab] report_fatal(VMErrorType, char const*, int,<br>
char const*, ...)+0x12b<br>
V [libjvm.so+0x67315d] report_java_out_of_memory(char const*)+0xed<br>
V [libjvm.so+0xc238d0]<br>
Metaspace::report_metadata_oome(ClassLoaderData*, unsigned long,<br>
MetaspaceObj::Type, Metaspace::MetadataType, JavaThread*)+0x260<br>
V [libjvm.so+0xc23b33] Metaspace::allocate(ClassLoaderData*,<br>
unsigned long, MetaspaceObj::Type, JavaThread*)+0x133<br>
V [libjvm.so+0x650e8b] ConstantPool::allocate(ClassLoaderData*, int,<br>
JavaThread*)+0x7b<br>
V [libjvm.so+0xae518c]<br>
VM_RedefineClasses::merge_cp_and_rewrite(InstanceKlass*,<br>
InstanceKlass*, JavaThread*)+0x4c<br>
V [libjvm.so+0xae6e6d] VM_RedefineClasses::load_new_class_versions()<br>
[clone .part.0]+0x34d<br>
V [libjvm.so+0xae794f] VM_RedefineClasses::doit_prologue()+0x17f<br>
V [libjvm.so+0xfa8ee0] VMThread::execute(VM_Operation*)+0x40<br>
V [libjvm.so+0xab416f] JvmtiEnv::RetransformClasses(int, _jclass*<br>
const*)+0x2bf<br>
V [libjvm.so+0xa5f04c] jvmti_RetransformClasses+0xfc<br>
C [libinstrument.so+0x4e86] retransformClasses+0x1b6<br>
J 182 sun.instrument.InstrumentationImpl.retransformClasses0(J[Ljava/lang/Class;)V<br>
java.instrument@20.0.1 (0 bytes) @ 0x00007f8d886d8077<br>
[0x00007f8d886d7fa0+0x00000000000000d7]<br>
J 181 c1 sun.instrument.InstrumentationImpl.retransformClasses([Ljava/lang/Class;)V<br>
java.instrument@20.0.1 (33 bytes) @ 0x00007f8d80c29a5c<br>
[0x00007f8d80c29860+0x00000000000001fc]<br>
j Agent.retransformLoop(Ljava/lang/instrument/Instrumentation;Ljava/lang/String;)V+82<br>
j Agent.lambda$premain$0(Ljava/lang/instrument/Instrumentation;Ljava/lang/String;)V+2<br>
j Agent$$Lambda$14+0x0000000801002c00.run()V+8<br>
j java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V+5<br>
java.base@20.0.1<br>
j java.lang.Thread.run()V+19 java.base@20.0.1<br>
v ~StubRoutines::call_stub 0x00007f8d88138cc6<br>
V [libjvm.so+0x8c9c85] JavaCalls::call_helper(JavaValue*,<br>
methodHandle const&, JavaCallArguments*, JavaThread*)+0x315<br>
V [libjvm.so+0x8cb5f2] JavaCalls::call_virtual(JavaValue*, Handle,<br>
Klass*, Symbol*, Symbol*, JavaThread*)+0x1d2<br>
V [libjvm.so+0x99debe] thread_entry(JavaThread*, JavaThread*)+0x8e<br>
V [libjvm.so+0x8e1578] JavaThread::thread_main_inner() [clone .part.0]+0xb8<br>
V [libjvm.so+0xf18736] Thread::call_run()+0xa6<br>
V [libjvm.so+0xcae8d8] thread_native_entry(Thread*)+0xd8<br>
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)<br>
J 182 sun.instrument.InstrumentationImpl.retransformClasses0(J[Ljava/lang/Class;)V<br>
java.instrument@20.0.1 (0 bytes) @ 0x00007f8d886d8004<br>
[0x00007f8d886d7fa0+0x0000000000000064]<br>
J 181 c1 sun.instrument.InstrumentationImpl.retransformClasses([Ljava/lang/Class;)V<br>
java.instrument@20.0.1 (33 bytes) @ 0x00007f8d80c29a5c<br>
[0x00007f8d80c29860+0x00000000000001fc]<br>
j Agent.retransformLoop(Ljava/lang/instrument/Instrumentation;Ljava/lang/String;)V+82<br>
j Agent.lambda$premain$0(Ljava/lang/instrument/Instrumentation;Ljava/lang/String;)V+2<br>
j Agent$$Lambda$14+0x0000000801002c00.run()V+8<br>
j java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V+5<br>
java.base@20.0.1<br>
j java.lang.Thread.run()V+19 java.base@20.0.1<br>
v ~StubRoutines::call_stub 0x00007f8d88138cc6<br>
<br>
I will continue to investigate on my side, but If you need more<br>
details, please ask.<br>
Jean-Philippe Bempel<br>
</blockquote></div>