C-heap memory leak of IceTea

[Zhu Han]

The program running is apache cassandra 0.6.8[1].  Based on the report of
cassandra community, most of them does not meet it even in their production
environment. Some of the users are big web company.

I have no idea whether sun jdk has same problem. Maybe I should have a try.

Do you have any hints on how to get more diagnostic information?

[1] http://cassandra.apache.org/

best regards,
hanzhu


On Mon, Dec 20, 2010 at 6:04 PM, Andrew Haley <aph at redhat.com> wrote:

> On 12/19/2010 02:59 AM, Zhu Han wrote:
>
>> Hi,
>>
>> I might observe C-heap leak of IcedTea6 1.9.2 on Ubuntu 10.04. This bug
>> was
>> reported against cassandra[1]. But I'm afraid it is a general openJDK
>> problem. Here is the symptoms. [1] may has more information on it.
>>
>> For a infrequently accessed openJDK node, the memory usage is climbed up
>> every day, and I'm pretty sure it is not caused by leak on Java object
>> heap.
>> This is the GC settings:
>>
>> " /usr/bin/java -ea -Xms1G -Xmx1G -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
>> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>> -XX:MaxTenuringThreshold=1
>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+HeapDumpOnOutOfMemoryError ..."
>>
>> So the size of JVM heap is constant. Use pmap against the JVM process for
>> the last 3 days, I observed a region expanded every day. It can be
>> expanded
>> to multiple gigabytes finally[1].
>>
>> on 12/19: 00000000010c3000 178548K rw---    [ anon ]
>> on 12/18: 00000000010c3000 110320K rw---    [ anon ]
>> on 12/17: 00000000010c3000  39256K rw---    [ anon ]
>>
>> I notice the region is on lower address of process address space. So I
>> suspect it is the c-heap of JVM. The map of object heap is as below, which
>> is constant in the last few days.
>>
>> 00002b58433c0000 1069824K rw---    [ anon ]
>>
>> Both the heap and non-heap memory usage of java code is quite normal:
>>
>> 12/19/2010 10:56:13 +0800 org.archive.jmx.Client HeapMemoryUsage:
>> committed: 1065025536
>> init: 1073741824
>> max: 1065025536
>> used: 160743000
>>
>> 12/19/2010 10:56:23 +0800 org.archive.jmx.Client NonHeapMemoryUsage:
>> committed: 24313856
>> init: 24313856
>> max: 224395264
>> used: 20477424
>>
>>
>> There are hundreds of alive threads in the JVM.
>>
>> 12/19/2010 10:52:20 +0800 org.archive.jmx.Client PeakThreadCount: 244
>>
>>
>>  Originally, I suspect it is bug 6824570[2]. But After I upgraded JDK from
>> IceTea 1.8.2 to IceTea 1.9.2, the problem is still not fixed. [3] said it
>> is
>> fixed in IceTea 1.9.0.
>>
>> And there is another JVM instances running on the same node with ParNewGC
>> enabled, I did not observe the memory leak.  Although the peak thread
>> count
>> is quite low of that instance, which is 43.
>>
>> Do you have any hints to diagnostic this problem? I'd like to help.
>>
>
> This may, of course, be a leak in JNI code.  What Java program was running?
>
> Andrew.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/distro-pkg-dev/attachments/20101220/8ffbf334/attachment.html