Heap OOM for no apparent reason?
Raman Gupta
rocketraman at fastmail.fm
Mon Jun 6 16:15:07 UTC 2011
To close this off from the perspective of the gc-dev list... using
jstack in combination with the OnOutOfMemoryError parameter was indeed
extremely useful in determining the allocation point causing the OOME.
The allocation point was an Object deserialization triggered by
Infinispan's unmarshalling code, and the OOME was swallowed by
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle()
by catching Throwable -- which only logs the Throwable if log.trace is
enabled (yuck).
In addition, at the point of the allocation failure the unmarshaller
is trying to allocate a very large array of chars which *should* be no
longer than 6, so basically it looks like there is likely a bug
somewhere in the application, in JGroups, or in Infinispan.
Thanks a lot to all of you for your help, as usual.
Cheers,
Raman Gupta
On 06/03/2011 11:50 AM, Y. Srinivas Ramakrishna wrote:
> John was right. I looked at the code and the process
> waits for the child to complete and return: the
> command is "safepoint-synchronous". May be all
> the j* commands are such that they can produce their
> results without needing mutator threads to run, because
> i also could not find any documented restrictions on the use of
> OnOutOfMemoryError.
>
> so, yes, looks like OnOutOfMemoryError will do what you want here,
> and my worries were baseless.
>
> thanks John!
> -- ramki
>
> On 6/3/2011 8:32 AM, Y. Srinivas Ramakrishna wrote:
>> By the way, it would seem that a "safepoint synchronous"
>> OnOutOfMemoryError execution would restrict what you could do,
>> just in case that caused a deadlock because the target (self)
>> might need to be at a non-safepoint to react to that command....
>>
>> Is there such a documented restriction on what commands can
>> be run within OnOutOfMemeoryError (or even a flat caveat emptor)?
>>
>> -- ramki
>>
>> On 6/3/2011 8:26 AM, Y. Srinivas Ramakrishna wrote:
>>> On 6/3/2011 7:52 AM, Raman Gupta wrote:
>>>> On 06/03/2011 03:28 AM, John Coomes wrote:
>>>>> Y. Srinivas Ramakrishna (y.s.ramakrishna at oracle.com) wrote:
>>>>>> Sorry, sent previous email without addressing all of the issues.
>>>>>>
>>>>>> On 6/2/2011 6:15 PM, Raman Gupta wrote:
>>>>>>> It would be *really* handy if there were a switch like:
>>>>>>>
>>>>>>> -XX:+StackTraceOnOutOfMemoryError
>>>>>>
>>>>>> Yes that would be handy, and probably not too difficult.
>>>>>> But I wonder also if something like OnOutOfMemoryError or
>>>>>> like would already get you enough info to get close to
>>>>>> the problem ... (although may be because it's executed in
>>>>>> a separate shell, by the time the command executes the
>>>>>> process has likely gone well past the point when the problem
>>>>>> occurred).
>>>>>
>>>>> No need to worry, the OnOutOfMemoryError commands are run while the
>>>>> JVM is at a safepoint. This worked for me:
>>>>>
>>>>> java -XX:OnOutOfMemoryError='jstack %p' ...
>>>
>>> Really, are you sure? I'd assumed you spawn off a separate (i.e.
>>> asynchronous)
>>> shell process rather than waiting for it to complete while you waited
>>> in the safepoint (i.e. synchronous). It could still be that one is
>>> "lucky" and the shell happens to complete before the safepoint is
>>> exited? Anyway, a good idea to check the code to see if there is
>>> a synchronicity guarantee or one relies on plain luck to sometimes
>>> get something useful (which itself is not bad, but good to know
>>> when it is good fortune vs actual design :-)
>>>
>>> -- ramki
>>>
>>>>>
>>>>> -John
>>>>
>>>> Excellent -- will try this later today.
>>>>
>>>> I did a quick search for places where OOME is caught and swallowed
>>>> and
>>>> found a few places within the JDK (such as direct ByteBuffer
>>>> allocation), as well as a couple places in other libraries such as
>>>> commons-pool and jgroups, the latter of which is used by Infinispan
>>>> (though in some cases, but not all, those are logged before being
>>>> swallowed). In short though, I still don't definitively know where
>>>> the
>>>> problem allocation is. So running jstack via OnOutOfMemoryError
>>>> sounds
>>>> like it is just the ticket.
>>>>
>>>> Cheers,
>>>> Raman
>>>
>>
>
More information about the hotspot-gc-dev
mailing list