A bug in C2 that causes a large amount of physical memory to be allocated
nijiaben
nijiaben at perfma.com
Fri Mar 15 10:18:45 UTC 2019
Hi, All
Recently, I‘m troubleshooting a JVM problem (JDK1.8.0_191-b12), and found that the process is always killed by the OS, which is caused by memory leaks. Finally, it was discovered that OOM is caused by a large amount of memory allocated by C2 thread. This is a bug in C2. The following is the troubleshooting process:
First, through /proc//smaps, I saw a lot of 64MB of memory allocation, and RSS is basically exhausted.
7fd690000000-7fd693f23000 rw-p 00000000 00:00 0 Size: 64652 kB Rss: 64652 kB Pss: 64652 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 64652 kB Referenced: 64652 kB Anonymous: 64652 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me nr sd 7fd693f23000-7fd694000000 ---p 00000000 00:00 0 Size: 884 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: mr mw me nr sd
Then trace the system call through the strace command, combined with the above virtual address, we found the corresponding mmap system call[pid 71] 13:34:41.982589 mmap(0x7fd690000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fd690000000 <0.000107>The thread that executes mmap is the 71 thread, so the thread is dumped through jstack, and the corresponding thread is found to be C2 CompilerThread0.
"C2 CompilerThread0" #39 daemon prio=9 os_prio=0 tid=0x00007fd8acebb000 nid=0x47 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE
Then grep the output of strace, see a lot of memory were allocated by this thread, probably more than 2G.
C2 is memory-limited under normal circumstances, but why is there a memory consumption greater than 2G? Finally, we found that a bug in the JVM will cause memory leaks. The location of this code is the nmethod::metadata_do method of nmethod.cpp:
void nmethod::metadata_do(void f(Metadata*)) { address low_boundary = verified_entry_point(); if (is_not_entrant()) { low_boundary += NativeJump::instruction_size; // %%% Note: On SPARC we patch only a 4-byte trap, not a full NativeJump. // (See comment above.) } { // Visit all immediate references that are embedded in the instruction stream. RelocIterator iter(this, low_boundary); while (iter.next()) { if (iter.type() == relocInfo::metadata_type ) { metadata_Relocation* r = iter.metadata_reloc(); // In this metadata, we must only follow those metadatas directly embedded in // the code. Other metadatas (oop_index>0) are seen as part of // the metadata section below. assert(1 == (r->metadata_is_immediate()) + (r->metadata_addr() >= metadata_begin() && r->metadata_addr() < metadata_end()), “metadata must be found in exactly one place”); if (r->metadata_is_immediate() && r->metadata_value() != NULL) { Metadata* md = r->metadata_value(); if (md != _method) f(md); } } else if (iter.type() == relocInfo::virtual_call_type) { // Check compiledIC holders associated with this nmethod CompiledIC *ic = CompiledIC_at(&iter); if (ic->is_icholder_call()) { CompiledICHolder* cichk = ic->cached_icholder(); f(cichk->holder_metadata()); f(cichk->holder_klass()); } else { Metadata* ic_oop = ic->cached_metadata(); if (ic_oop != NULL) { f(ic_oop); } } } } }
Because CompiledIC is a ResourceObj, but it is not freed by ResourceMark, so when this method is called multiple times, OOM will appear.
The fix is very simple, just add ResourceMark rm; before CompiledIC *ic = CompiledIC_at(&iter);
And I found out that it has been solved on JDK12.
http://hg.openjdk.java.net/jdk-updates/jdk12u/rev/aa3bfacc912c
Can this patch be entered into JDK8?
Thanks,
nijiaben @ PerfMa
More information about the jdk8u-dev
mailing list