C-heap memory leak of IceTea

[Andrew Haley]

On 12/22/2010 05:12 AM, Zhu Han wrote:
> By the way, cassandra use JNA to invoke native APIs. Is there any know
> problem of JNA over openJDK?

No.  But I suspect your native code is leaking memory.

> On Mon, Dec 20, 2010 at 6:04 PM, Andrew Haley<aph at redhat.com>  wrote:
>
>> On 12/19/2010 02:59 AM, Zhu Han wrote:
>>
>>> Hi,
>>>
>>> I might observe C-heap leak of IcedTea6 1.9.2 on Ubuntu 10.04. This bug
>>> was
>>> reported against cassandra[1]. But I'm afraid it is a general openJDK
>>> problem. Here is the symptoms. [1] may has more information on it.
>>>
>>> For a infrequently accessed openJDK node, the memory usage is climbed up
>>> every day, and I'm pretty sure it is not caused by leak on Java object
>>> heap.
>>> This is the GC settings:
>>>
>>> " /usr/bin/java -ea -Xms1G -Xmx1G -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
>>> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>>> -XX:MaxTenuringThreshold=1
>>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>>> -XX:+HeapDumpOnOutOfMemoryError ..."
>>>
>>> So the size of JVM heap is constant. Use pmap against the JVM process for
>>> the last 3 days, I observed a region expanded every day. It can be
>>> expanded
>>> to multiple gigabytes finally[1].
>>>
>>> on 12/19: 00000000010c3000 178548K rw---    [ anon ]
>>> on 12/18: 00000000010c3000 110320K rw---    [ anon ]
>>> on 12/17: 00000000010c3000  39256K rw---    [ anon ]
>>>
>>> I notice the region is on lower address of process address space. So I
>>> suspect it is the c-heap of JVM. The map of object heap is as below, which
>>> is constant in the last few days.
>>>
>>> 00002b58433c0000 1069824K rw---    [ anon ]
>>>
>>> Both the heap and non-heap memory usage of java code is quite normal:
>>>
>>> 12/19/2010 10:56:13 +0800 org.archive.jmx.Client HeapMemoryUsage:
>>> committed: 1065025536
>>> init: 1073741824
>>> max: 1065025536
>>> used: 160743000
>>>
>>> 12/19/2010 10:56:23 +0800 org.archive.jmx.Client NonHeapMemoryUsage:
>>> committed: 24313856
>>> init: 24313856
>>> max: 224395264
>>> used: 20477424
>>>
>>>
>>> There are hundreds of alive threads in the JVM.
>>>
>>> 12/19/2010 10:52:20 +0800 org.archive.jmx.Client PeakThreadCount: 244
>>>
>>>
>>>   Originally, I suspect it is bug 6824570[2]. But After I upgraded JDK from
>>> IceTea 1.8.2 to IceTea 1.9.2, the problem is still not fixed. [3] said it
>>> is
>>> fixed in IceTea 1.9.0.
>>>
>>> And there is another JVM instances running on the same node with ParNewGC
>>> enabled, I did not observe the memory leak.  Although the peak thread
>>> count
>>> is quite low of that instance, which is 43.
>>>
>>> Do you have any hints to diagnostic this problem? I'd like to help.
>>>
>>
>> This may, of course, be a leak in JNI code.  What Java program was running?
>>
>> Andrew.
>>
>