CodeHeap::expand_by malloc failed
Vitaly Davidovich
vitalyd at gmail.com
Mon Jun 2 15:26:30 UTC 2014
Hi Stefan,
Good find. I checked, and the max map count is set to 1M, and looking at
the map output in the error file, I'm pretty sure we're not even close to
that (or 65k for that matter) :). I'll keep digging for now ...
Thanks
On Mon, Jun 2, 2014 at 10:03 AM, Stefan Karlsson <stefan.karlsson at oracle.com
> wrote:
>
> On 2014-05-30 17:04, Vitaly Davidovich wrote:
>
>> Hi Dan,
>>
>> Thanks for the pointer. Here's the relevant line:
>>
>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>> os::commit_memory(0x00002b091869c000, 65536, 1) failed; error='Cannot
>> allocate memory' (errno=12)
>>
>> So it confirms the fixed address at which it's trying to commit and the
>> 65kb size. The address matches the end of the currently committed code
>> heap region, but still somewhat puzzling as to why ENOMEM is returned
>> given
>> the rest of the output in the error report. Perhaps there was a transient
>> spike in mem consumption when mmap failed but the condition disappeared
>> (ie. mem was free'd back to os by another process) by the time the error
>> reporter pulled mem stats from the kernel?
>>
>
> man mmap says:
>
> ENOMEM No memory is available, or the process's maximum number of
> mappings would have been exceeded.
>
>
> Could you have exceeded the maximum number of vma mappings?
>
> The crash happens when we are converting reserved memory into committed
> memory. Among other things, this will cause a vma split and from mmap.c:
>
> int split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
> unsigned long addr, int new_below)
> {
> if (mm->map_count >= sysctl_max_map_count)
> return -ENOMEM;
>
> return __split_vma(mm, vma, addr, new_below);
> }
>
> This seems to be the default value:
> $ sysctl vm/max_map_count
> vm.max_map_count = 65530
>
> StefanK
>
>
>
>> Thanks
>>
>>
>> On Fri, May 30, 2014 at 10:14 AM, Daniel D. Daugherty <
>> daniel.daugherty at oracle.com> wrote:
>>
>> When 'os::commit_memory()' fails, there should be a message like the
>>> following in stderr for the Java process:
>>>
>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(
>>> 0xfffffd7fe8c00000,
>>> 2097152, 1) failed; error='Resource temporarily unavailable' (errno=11)
>>>
>>> The address and size should match what you see in the stack trace
>>> which provides confirmation that we have the right mesg, but the
>>> important piece is the errno value...
>>>
>>> Dan
>>>
>>>
>>>
>>> On 5/29/14 7:20 PM, Vitaly Davidovich wrote:
>>>
>>> Yes, swap and overcommit are turned off. But, there's substantial free
>>>> physical memory available and looks like 65kb available to expand the
>>>> code
>>>> heap contiguoisly. So I'm a bit puzzled ...
>>>>
>>>> Taking Chris' suggestion and cc'ing hotspot dev ...
>>>>
>>>> Sent from my phone
>>>> On May 29, 2014 8:47 PM, "Vladimir Kozlov" <vladimir.kozlov at oracle.com>
>>>> wrote:
>>>>
>>>> Could be because there is no swap on this machine:
>>>>
>>>>> Memory: 4k page, physical 49521668k(5596796k free), swap 0k(0k free)
>>>>>
>>>>> Vladimir
>>>>>
>>>>> On 5/29/14 5:35 PM, Christian Thalinger wrote:
>>>>>
>>>>> Although it’s the code cache I assume runtime folk would know more
>>>>> about
>>>>>
>>>>>> this. Maybe send to hotspot-dev.
>>>>>>
>>>>>> On May 29, 2014, at 9:01 AM, Vitaly Davidovich <vitalyd at gmail.com
>>>>>> <mailto:vitalyd at gmail.com>> wrote:
>>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> Need a bit of help explaining a hotspot malloc failure crash on 7u51.
>>>>>>> I'm going to paste the relevant snippets from the hs_err log.
>>>>>>>
>>>>>>> #
>>>>>>> # There is insufficient memory for the Java Runtime Environment to
>>>>>>> continue.
>>>>>>> # Native memory allocation (malloc) failed to allocate 65536 bytes
>>>>>>> for
>>>>>>> committing reserved memory.
>>>>>>> # Possible reasons:
>>>>>>> # The system is out of physical RAM or swap space
>>>>>>> # In 32 bit mode, the process size limit was hit
>>>>>>> # Possible solutions:
>>>>>>> # Reduce memory load on the system
>>>>>>> # Increase physical memory or swap space
>>>>>>> # Check if swap backing store is full
>>>>>>> # Use 64 bit Java on a 64 bit OS
>>>>>>> # Decrease Java heap size (-Xmx/-Xms)
>>>>>>> # Decrease number of Java threads
>>>>>>> # Decrease Java thread stack sizes (-Xss)
>>>>>>> # Set larger code cache with -XX:ReservedCodeCacheSize=
>>>>>>> # This output file may be truncated or incomplete.
>>>>>>> #
>>>>>>> # Out of Memory Error (os_linux.cpp:2726), pid=10643,
>>>>>>> tid=47319048501520
>>>>>>> #
>>>>>>> # JRE version: Java(TM) SE Runtime Environment (7.0_51-b13) (build
>>>>>>> 1.7.0_51-b13)
>>>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode
>>>>>>> linux-amd64 compressed oops)
>>>>>>>
>>>>>>>
>>>>>>> --------------- T H R E A D ---------------
>>>>>>>
>>>>>>> Current thread (0x0000000000716800): JavaThread "C2 CompilerThread0"
>>>>>>> daemon [_thread_in_vm, id=10689,
>>>>>>> stack(0x00002b095303b000,0x00002b095313c000)]
>>>>>>>
>>>>>>> Stack: [0x00002b095303b000,0x00002b095313c000],
>>>>>>> sp=0x00002b0953138d20, free space=1015k
>>>>>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
>>>>>>> C=native code)
>>>>>>> V [libjvm.so+0x992f4a] VMError::report_and_die()+0x2ea
>>>>>>> V [libjvm.so+0x4931ab] report_vm_out_of_memory(char const*, int,
>>>>>>> unsigned long, char const*)+0x9b
>>>>>>> V [libjvm.so+0x81338e] os::Linux::commit_memory_impl(char*,
>>>>>>> unsigned
>>>>>>> long, bool)+0xfe
>>>>>>> V [libjvm.so+0x81383f] os::Linux::commit_memory_impl(char*,
>>>>>>> unsigned
>>>>>>> long, unsigned long, bool)+0x4f
>>>>>>> V [libjvm.so+0x813a2c] os::pd_commit_memory(char*, unsigned long,
>>>>>>> unsigned long, bool)+0xc
>>>>>>> V [libjvm.so+0x80daea] os::commit_memory(char*, unsigned long,
>>>>>>> unsigned long, bool)+0x2a
>>>>>>> V [libjvm.so+0x98e849] VirtualSpace::expand_by(unsigned long,
>>>>>>> bool)+0x1c9
>>>>>>> V [libjvm.so+0x58a62c] CodeHeap::expand_by(unsigned long)+0x8c
>>>>>>> V [libjvm.so+0x42111d] CodeCache::allocate(int)+0x4d
>>>>>>> V [libjvm.so+0x7e1a39] nmethod::new_nmethod(methodHandle, int,
>>>>>>> int,
>>>>>>> CodeOffsets*, int, DebugInformationRecorder*, Dependencies*,
>>>>>>> CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*,
>>>>>>> ImplicitExceptionTable*, AbstractCompiler*, int)+0x179
>>>>>>> V [libjvm.so+0x3cfc54] ciEnv::register_method(ciMethod*, int,
>>>>>>> CodeOffsets*, int, CodeBuffer*, int, OopMapSet*,
>>>>>>> ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*,
>>>>>>> int, bool, bool)+0x364
>>>>>>> V [libjvm.so+0x4458fb] Compile::Compile(ciEnv*, C2Compiler*,
>>>>>>> ciMethod*, int, bool, bool)+0x11cb
>>>>>>> V [libjvm.so+0x3afa76] C2Compiler::compile_method(ciEnv*,
>>>>>>> ciMethod*,
>>>>>>> int)+0x176
>>>>>>> V [libjvm.so+0x44ba9e]
>>>>>>> CompileBroker::invoke_compiler_on_method(CompileTask*)+0x33e
>>>>>>> V [libjvm.so+0x44c87d] CompileBroker::compiler_thread_loop()+0x43d
>>>>>>> V [libjvm.so+0x94d5ff] JavaThread::thread_main_inner()+0xdf
>>>>>>> V [libjvm.so+0x94d705] JavaThread::run()+0xf5
>>>>>>> V [libjvm.so+0x815538] java_start(Thread*)+0x108
>>>>>>>
>>>>>>> Code Cache [0x00002b091755c000, 0x00002b091869c000,
>>>>>>> 0x00002b091a55c000)
>>>>>>> total_blobs=7033 nmethods=6349 adapters=634
>>>>>>> free_code_cache=31715Kb
>>>>>>> largest_free_block=32247296
>>>>>>>
>>>>>>> Memory mappings around the code cache virtual space (bolded line is
>>>>>>> the code cache segment, I believe, based on the Code Cache output
>>>>>>> above):
>>>>>>>
>>>>>>> *2b091755c000-2b091869c000 rwxp 00000000 00:00 0*
>>>>>>> 2b09186ac000-2b091d66f000 rw-p 00000000 00:00 0
>>>>>>> 2b091d66f000-2b091d670000 ---p 00000000 00:00 0
>>>>>>> 2b091d670000-2b091d770000 rwxp 00000000 00:00 0
>>>>>>> 2b091d770000-2b091d771000 ---p 00000000 00:00 0
>>>>>>> 2b091d771000-2b091d871000 rwxp 00000000 00:00 0
>>>>>>> 2b091d871000-2b091d872000 ---p 00000000 00:00 0
>>>>>>> 2b091d872000-2b091d972000 rwxp 00000000 00:00 0
>>>>>>>
>>>>>>> /proc/meminfo:
>>>>>>> MemTotal: 49521668 kB
>>>>>>> MemFree: 5596796 kB
>>>>>>> Buffers: 294684 kB
>>>>>>> Cached: 34205856 kB
>>>>>>> SwapCached: 0 kB
>>>>>>> Active: 12745128 kB
>>>>>>> Inactive: 28516788 kB
>>>>>>> Active(anon): 9280636 kB
>>>>>>> Inactive(anon): 3090264 kB
>>>>>>> Active(file): 3464492 kB
>>>>>>> Inactive(file): 25426524 kB
>>>>>>> Unevictable: 14420 kB
>>>>>>> Mlocked: 14420 kB
>>>>>>> SwapTotal: 0 kB
>>>>>>> SwapFree: 0 kB
>>>>>>> Dirty: 20392 kB
>>>>>>> Writeback: 0 kB
>>>>>>> AnonPages: 6776540 kB
>>>>>>> Mapped: 6292204 kB
>>>>>>> Shmem: 5604620 kB
>>>>>>> Slab: 1828656 kB
>>>>>>> SReclaimable: 1567928 kB
>>>>>>> SUnreclaim: 260728 kB
>>>>>>> KernelStack: 4648 kB
>>>>>>> PageTables: 58800 kB
>>>>>>> NFS_Unstable: 0 kB
>>>>>>> Bounce: 0 kB
>>>>>>> WritebackTmp: 0 kB
>>>>>>> CommitLimit: 47045584 kB
>>>>>>> Committed_AS: 46046480 kB
>>>>>>> VmallocTotal: 34359738367 kB
>>>>>>> VmallocUsed: 460188 kB
>>>>>>> VmallocChunk: 34333703480 kB
>>>>>>> HugePages_Total: 0
>>>>>>> HugePages_Free: 0
>>>>>>> HugePages_Rsvd: 0
>>>>>>> HugePages_Surp: 0
>>>>>>> Hugepagesize: 2048 kB
>>>>>>> DirectMap4k: 8192 kB
>>>>>>> DirectMap2M: 2080768 kB
>>>>>>> DirectMap1G: 48234496 kB
>>>>>>>
>>>>>>> Memory: 4k page, physical 49521668k(5596796k free), swap 0k(0k free)
>>>>>>>
>>>>>>> * overcommit is turned off
>>>>>>>
>>>>>>> So the code heap was attempting to expand by 65kb at a fixed address.
>>>>>>> There appears to be a 65kb mapping available between the end of
>>>>>>> the
>>>>>>> current code heap mapping (2b091755c000-2b091869c000) and the next
>>>>>>> one
>>>>>>> (2b09186ac000-2b091d66f000). There's about 30mb of free space left
>>>>>>> (reserved, but uncommitted I take it) in the virtual space, so it's a
>>>>>>> bit puzzling given there's ample physical mem available. Only thing
>>>>>>> I
>>>>>>> can think of is expansion fails because it cannot get contiguous
>>>>>>> space, but then I can't reconcile that with the mem mapping above.
>>>>>>>
>>>>>>> Does anyone have any ideas?
>>>>>>>
>>>>>>> Let me know if you need additional info from the hs_err file.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/attachments/20140602/2d6278e1/attachment-0001.html>
More information about the hotspot-compiler-dev
mailing list