Fwd: GC overhead limit exceeded

[Hannes Wallnoefer]

Am 2014-03-06 14:23, schrieb Tal Liron:
> Have you looked at the dump I linked to before?
>
> https://dl.dropboxusercontent.com/u/122806/jvm8_gc2.zip

Yes, this is the dump I looked at.

> Of course it could be a memory leak in my code, but by far the largest 
> amount of instances are held by lambda forms. It could be that they 
> take very little memory, so it might not be important. However, I 
> understood that this is actually a known issue in JVM 7, too. But I 
> don't know enough about how Nashorn uses it to comment.

Lambda forms are used by Nashorn internally to link callsites at 
runtime. However, even if it is lambda forms that are leaked that does 
not mean the problem must be within Nashorn (I'm not saying the opposite 
either - just trying to find out what's going on).

The ThreadLocal thing could be a Nashorn issue as we keep the current 
global object in a ThreadLocal. Are you keeping a lot of threads alive 
after they interact with Nashorn (e.g. run some script)?

The other trace I'd follow is the TimerThread -> PrudenceApplication 
path. There seems to be a lot of stuff (Nashorn related and else) stored 
in ConcurrentHashMaps stored in the context of PrudenceApplication. As I 
said I don't understand enough to say whether that's the way it's meant 
to be; you're probably in a better position to tell.

Hannes

> On 03/06/2014 09:17 PM, Hannes Wallnoefer wrote:
>> Hi Tal,
>>
>> I'm sorry this problem still persists.
>>
>> Back when I tried your app I threw a lot of apache bench requests at 
>> it but didn't see a leak. Looking at your dump again I notice that 
>> while there are a lot of LambdaForm related instances they don't 
>> occupy a large part of the heap.
>>
>> The biggest part of the heap (over 40 %) is used for character 
>> arrays, mostly from Strings. A lot of them are GC-rooted in 
>> TimerThread or ThreadLocal instances. I don't understand the 
>> application well enough to know whether that is the root of the 
>> problem. Could it be that you're retaining some reference in some 
>> recurring/scheduler tasks you're running?
>>
>> Hannes
>>
>> Am 2014-03-06 07:28, schrieb Tal Liron:
>>> Well, I've reached the limits of my personal knowledge to work on 
>>> this problem. I'm happy to assist in any way I can, including 
>>> providing access to servers.
>>>
>>> As it stands, however, it seems that this problem will follow 
>>> through into the official release of OpenJDK 8, which means that I 
>>> won't be able to recommend Nashorn for production deployments of 
>>> Prudence. Perhaps there will be a fix later on?
>>>
>>> (By the way, there is no bug with Log4j 2.0 -- the high number of 
>>> instances is by design, an effect of using the innovative Disruptor 
>>> library for inter-thread communication.)
>>>
>>> On 02/25/2014 02:12 AM, Tal Liron wrote:
>>>> I've been away for a month. Has anyone with knowhow followed up on 
>>>> this?
>>>>
>>>> The issue is still present.
>>>>
>>>> On 01/18/2014 02:51 PM, Tal Liron wrote:
>>>>> I have a new dump that will hopefully be more useful:
>>>>>
>>>>> https://dl.dropboxusercontent.com/u/122806/jvm8_gc2.zip
>>>>>
>>>>> From what I can tell, indeed lambda forms are way out of control 
>>>>> here. Generally, too, there is a huge amount of Nashorn-related 
>>>>> instances, which may be related.
>>>>>
>>>>> (Note that Log4j 2.0 also seems to be having a serious memory 
>>>>> leak! I have opened a bug about it over there.)
>>>
>>
>