CodeHeap::expand_by malloc failed

Thu May 29 16:01:16 UTC 2014

Hi guys,

Need a bit of help explaining a hotspot malloc failure crash on 7u51.  I'm
going to paste the relevant snippets from the hs_err log.

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 65536 bytes for
committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2726), pid=10643, tid=47319048501520
#
# JRE version: Java(TM) SE Runtime Environment (7.0_51-b13) (build
1.7.0_51-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode
linux-amd64 compressed oops)

---------------  T H R E A D  ---------------

Current thread (0x0000000000716800):  JavaThread "C2 CompilerThread0"
daemon [_thread_in_vm, id=10689,
stack(0x00002b095303b000,0x00002b095313c000)]

Stack: [0x00002b095303b000,0x00002b095313c000],  sp=0x00002b0953138d20,
 free space=1015k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
V  [libjvm.so+0x992f4a]  VMError::report_and_die()+0x2ea
V  [libjvm.so+0x4931ab]  report_vm_out_of_memory(char const*, int, unsigned
long, char const*)+0x9b
V  [libjvm.so+0x81338e]  os::Linux::commit_memory_impl(char*, unsigned
long, bool)+0xfe
V  [libjvm.so+0x81383f]  os::Linux::commit_memory_impl(char*, unsigned
long, unsigned long, bool)+0x4f
V  [libjvm.so+0x813a2c]  os::pd_commit_memory(char*, unsigned long,
unsigned long, bool)+0xc
V  [libjvm.so+0x80daea]  os::commit_memory(char*, unsigned long, unsigned
long, bool)+0x2a
V  [libjvm.so+0x98e849]  VirtualSpace::expand_by(unsigned long, bool)+0x1c9
V  [libjvm.so+0x58a62c]  CodeHeap::expand_by(unsigned long)+0x8c
V  [libjvm.so+0x42111d]  CodeCache::allocate(int)+0x4d
V  [libjvm.so+0x7e1a39]  nmethod::new_nmethod(methodHandle, int, int,
CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*,
int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*,
AbstractCompiler*, int)+0x179
V  [libjvm.so+0x3cfc54]  ciEnv::register_method(ciMethod*, int,
CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*,
ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool)+0x364
V  [libjvm.so+0x4458fb]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*,
int, bool, bool)+0x11cb
V  [libjvm.so+0x3afa76]  C2Compiler::compile_method(ciEnv*, ciMethod*,
int)+0x176
V  [libjvm.so+0x44ba9e]
 CompileBroker::invoke_compiler_on_method(CompileTask*)+0x33e
V  [libjvm.so+0x44c87d]  CompileBroker::compiler_thread_loop()+0x43d
V  [libjvm.so+0x94d5ff]  JavaThread::thread_main_inner()+0xdf
V  [libjvm.so+0x94d705]  JavaThread::run()+0xf5
V  [libjvm.so+0x815538]  java_start(Thread*)+0x108

Code Cache  [0x00002b091755c000, 0x00002b091869c000, 0x00002b091a55c000)
 total_blobs=7033 nmethods=6349 adapters=634 free_code_cache=31715Kb
largest_free_block=32247296

Memory mappings around the code cache virtual space (bolded line is the
code cache segment, I believe, based on the Code Cache output above):

*2b091755c000-2b091869c000 rwxp 00000000 00:00 0*
2b09186ac000-2b091d66f000 rw-p 00000000 00:00 0
2b091d66f000-2b091d670000 ---p 00000000 00:00 0
2b091d670000-2b091d770000 rwxp 00000000 00:00 0
2b091d770000-2b091d771000 ---p 00000000 00:00 0
2b091d771000-2b091d871000 rwxp 00000000 00:00 0
2b091d871000-2b091d872000 ---p 00000000 00:00 0
2b091d872000-2b091d972000 rwxp 00000000 00:00 0

/proc/meminfo:
MemTotal:       49521668 kB
MemFree:         5596796 kB
Buffers:          294684 kB
Cached:         34205856 kB
SwapCached:            0 kB
Active:         12745128 kB
Inactive:       28516788 kB
Active(anon):    9280636 kB
Inactive(anon):  3090264 kB
Active(file):    3464492 kB
Inactive(file): 25426524 kB
Unevictable:       14420 kB
Mlocked:           14420 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:             20392 kB
Writeback:             0 kB
AnonPages:       6776540 kB
Mapped:          6292204 kB
Shmem:           5604620 kB
Slab:            1828656 kB
SReclaimable:    1567928 kB
SUnreclaim:       260728 kB
KernelStack:        4648 kB
PageTables:        58800 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    47045584 kB
Committed_AS:   46046480 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      460188 kB
VmallocChunk:   34333703480 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8192 kB
DirectMap2M:     2080768 kB
DirectMap1G:    48234496 kB

Memory: 4k page, physical 49521668k(5596796k free), swap 0k(0k free)

* overcommit is turned off

So the code heap was attempting to expand by 65kb at a fixed address.
 There appears to be a 65kb mapping available between the end of the
current code heap mapping (2b091755c000-2b091869c000) and the next one
(2b09186ac000-2b091d66f000).  There's about 30mb of free space left
(reserved, but uncommitted I take it) in the virtual space, so it's a bit
puzzling given there's ample physical mem available.  Only thing I can
think of is expansion fails because it cannot get contiguous space, but
then I can't reconcile that with the mem mapping above.

Does anyone have any ideas?

Let me know if you need additional info from the hs_err file.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/attachments/20140529/37fdf290/attachment-0001.html>