Debugging Resident Set Size growth

[Chris Burroughs]

Hello all.  I am experiencing a problem with a java program on running
1.6.0_23 on RHEL5.  The resident set size (as reported by
/proc/$PID/status for example) of the process continues to grow.  For
example here [1] is the RES of the process from approximately the 24th
to 72nd hours of operation with min and max heap equal to 1.5 GB.  This
corresponds to a an decrease in "-/+ buffers/cache free" memory as
reported by `free`.  Jconsole reports that the max heap limit is being
respected.  This continues over a few weeks until their is so little
page cache that performance degrades or the kernel oom killer steps in.

The program (Apache Cassandra) is not configured to use mmap or as far
as I can tell anything improper with direct allocation.   I understand
that the JVM has internal structures that take memory, that GC can only
periodically compact the heap, etc.  However, unless I am missing
something the RES should eventually (say within days) come down.  This
is the case with every other java program I have monitored, but again
unless I'm missing something it should not be possible for any Java
program to induce this different behaviour.

Since bugs in the JVM or libc level seem unlikely I have been searching
for a document along the lines of "you are probaly wrong, here are all
the things you should do before claiming you found a memory leak in a
JVM", but have been unable to find one.  The "Java Trouble-Shooting and
Diagnostic Guide" contains section on OOM Exceptions and JIN, but not
hotspot itself.  If someone could point me in the direction of one I
would appreciate it.

Short of that are there steps to make a useful bug report short of "run
the program with valgrind for n days"?

Thank you for your time,
Chris Burroughs

[1] http://img16.imageshack.us/img16/4813/cassandrarsssmooth.png