Debugging Resident Set Size growth

[Y. S. Ramakrishna]

Can you use pmap -x to see where and what the growth is in?
For example, if your Java heap needs increase then the
committed part of the heap will become resident as and
when those pages are actually touched or used. So
with a program that eventually grows to occupy the heap
it's been given, one would eventually expect to reach 1.5 G
and some more (for the perm gen, the C-heap and so forth).
I am assuming that the unit on the y-axis of your graph is
KB, so that at the right end of your graph you are
reaching an RSS of a bit over 2 G. (I also assume from
you description that your x-axis is about 72 hours, although
there are no labels.)

Did you look at the GC logs or at the Java heap with jmap or jconsole?

In summary, without more detaiol, I do not see that there is
necessarily a problem here to solve (although i do sympathise
with the Linux OOM killer, I am sure it will not step in
just because your RSS is 2 GB for a program that you seek
to give 1.5 GB of java heap; how much RAM does your machine
have?)

-- ramki

On 04/13/11 18:30, Chris Burroughs wrote:
> Hello all.  I am experiencing a problem with a java program on running
> 1.6.0_23 on RHEL5.  The resident set size (as reported by
> /proc/$PID/status for example) of the process continues to grow.  For
> example here [1] is the RES of the process from approximately the 24th
> to 72nd hours of operation with min and max heap equal to 1.5 GB.  This
> corresponds to a an decrease in "-/+ buffers/cache free" memory as
> reported by `free`.  Jconsole reports that the max heap limit is being
> respected.  This continues over a few weeks until their is so little
> page cache that performance degrades or the kernel oom killer steps in.
> 
> The program (Apache Cassandra) is not configured to use mmap or as far
> as I can tell anything improper with direct allocation.   I understand
> that the JVM has internal structures that take memory, that GC can only
> periodically compact the heap, etc.  However, unless I am missing
> something the RES should eventually (say within days) come down.  This
> is the case with every other java program I have monitored, but again
> unless I'm missing something it should not be possible for any Java
> program to induce this different behaviour.
> 
> Since bugs in the JVM or libc level seem unlikely I have been searching
> for a document along the lines of "you are probaly wrong, here are all
> the things you should do before claiming you found a memory leak in a
> JVM", but have been unable to find one.  The "Java Trouble-Shooting and
> Diagnostic Guide" contains section on OOM Exceptions and JIN, but not
> hotspot itself.  If someone could point me in the direction of one I
> would appreciate it.
> 
> Short of that are there steps to make a useful bug report short of "run
> the program with valgrind for n days"?
> 
> Thank you for your time,
> Chris Burroughs
> 
> [1] http://img16.imageshack.us/img16/4813/cassandrarsssmooth.png