Debugging Resident Set Size growth

Mon May 2 09:56:17 PDT 2011

Hello,

Haven't seen any recent replies to this but wanted to send a note as
I'm looking into a similar issue. We have witnessed the behavior of
the java process eating through all available memory until the
oom-killer steps in using both 6u18 and now 6u23.

Our environment looks like:

* Large amazon EC2 instance, 7.5GB RAM / 64bit
* OS - Linux, CentOS release 5.4 final
* Kernel version 2.6.21
* Apache tomcat is the only java process on the machine. Based on
reports the RES process has been seen at 6.5GB+ before the oom-killer
comes along.

The JVM args for this process are:

-server
-Djavax.net.ssl.trustStore=$JAVA_HOME/jre/lib/security/cacerts
-Djavax.net.ssl.trustStorePassword=changeit
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=XXX.XXX.XXX
-Dcom.sun.management.snmp.interface=0.0.0.0
-Dcom.sun.management.snmp.port=9161
-Dcom.sun.management.snmp.acl.file=/usr/java/latest/jre/lib/management/snmp.acl
-Dsun.net.inetaddr.ttl=-1
-XX:+UseConcMarkSweepGC
-Xmx2G
-XX:NewSize=512m
-XX:MaxNewSize=512m
-XX:MaxPermSize=512m
-XX:-TraceClassUnloading
-Xrunjdwp:transport=dt_socket,address=9002,server=y,suspend=n
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/usr/local/projects/logs/heap_dumps
-Xloggc:/usr/local/projects/logs/gc/gc-${LOG_TIME}.log
-XX:+PrintGCDetails
-verbose:gc
-XX:CMSInitiatingOccupancyFraction=50
-XX:+DisableExplicitGC
-agentpath:/XXX/lib/libyjpagent.so=port=9000

I had been taking pmap samples about once a day, and have 10
collection outputs. The pmap samples started after I restarted tomcat,
and ended when tomcat needed to be brought down for other reasons. At
the time of take down, I believe the RES size was ~5.3GB

One other thing to note is, the above configuration is what we use for
North America which are our highest usage production instances. We
have a couple instances in Amazon EC2's Europe and APAC zones, which
are "Small" instances. These instances have the same setup, use
tomcat, same applications deployed, etc but have smaller heap sizes
(1.2gb) due to less resources available. These instances are also
32bit rather than 64bit. We do not seem to experience the constant
consumption of RES memory on our 32bit machines as we do on the 64 bit
ones. I'm not sure if we aren't seeing the same results simply due to
large differences in usage.

I'v attached the pmap output iv collected. Any help on figuring this
out would be appreciated, please let me know if you need anything
further.

Thanks
Chris

On Fri, Apr 15, 2011 at 2:52 PM, Chris Burroughs
<chris.burroughs at gmail.com> wrote:
> Thanks you for your suggestions.  I didn't think to use pmap before.
>
> On 04/13/2011 09:55 PM, Y. S. Ramakrishna wrote:
>> Can you use pmap -x to see where and what the growth is in?
>
> Individual calls to pmap as of right now.  Will look into measuring
> changes over time.
>
> http://pastebin.com/G0SQ7mQQ
> http://pastebin.com/ARPF8gPk
>
>>
>> Did you look at the GC logs or at the Java heap with jmap or jconsole?
>>
>
> I don't see GC large looking pauses dangerous looking GC pressure, or
> running close to max heap size from logs or jconsole.
>
>> In summary, without more detaiol, I do not see that there is
>> necessarily a problem here to solve (although i do sympathise
>> with the Linux OOM killer, I am sure it will not step in
>> just because your RSS is 2 GB for a program that you seek
>> to give 1.5 GB of java heap; how much RAM does your machine
>> have?)
>>
>
> Box has 4 GB of RAM, 2 GB swap.  There is another .5 gig java (tomcat)
> process.
>
>
> As requested I'll move further discussion here:
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7037080
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2011-04-18:23.14.56
Type: application/octet-stream
Size: 26122 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110502/f942910b/attachment-0010.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2011-04-19:22.18.12
Type: application/octet-stream
Size: 26256 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110502/f942910b/attachment-0011.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2011-04-20:22.50.28
Type: application/octet-stream
Size: 26323 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110502/f942910b/attachment-0012.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2011-04-21:22.08.27
Type: application/octet-stream
Size: 30085 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110502/f942910b/attachment-0013.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2011-04-22:22.41.12
Type: application/octet-stream
Size: 30152 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110502/f942910b/attachment-0014.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2011-04-24:14.24.41
Type: application/octet-stream
Size: 30152 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110502/f942910b/attachment-0015.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2011-04-24:22.44.19
Type: application/octet-stream
Size: 30152 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110502/f942910b/attachment-0016.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2011-04-26:09.45.35
Type: application/octet-stream
Size: 34440 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110502/f942910b/attachment-0017.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2011-04-27:00.11.03
Type: application/octet-stream
Size: 34842 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110502/f942910b/attachment-0018.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2011-04-27:21.36.24
Type: application/octet-stream
Size: 35914 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20110502/f942910b/attachment-0019.obj 