Debugging Resident Set Size growth

[Coleen Phillimore]

Hi,
We had another unverifiable sighting of this sort, so I'd like to follow 
up and see if we can figure out what is going on.   A couple of questions:

1. Was this not a problem with earlier releases of jdk6 on RHEL5?  Did 
this just start with 1.6.0_23?

2.  Running with valgrind would be a good idea.  It has been used to 
find leaks in the VM in the past (6980262) but the VM mostly uses fixed 
memory in the permgen for internal VM structures and other fixed size 
arenas, so the leak may also be in the jdk native libraries or the 
application native libraries.

This isn't really a hotspot open source development question, can you 
file a bug?
  http://java.sun.com/webapps/bugreport/crash.jsp

Thanks,
Coleen

On 4/13/2011 9:30 PM, Chris Burroughs wrote:
> Hello all.  I am experiencing a problem with a java program on running
> 1.6.0_23 on RHEL5.  The resident set size (as reported by
> /proc/$PID/status for example) of the process continues to grow.  For
> example here [1] is the RES of the process from approximately the 24th
> to 72nd hours of operation with min and max heap equal to 1.5 GB.  This
> corresponds to a an decrease in "-/+ buffers/cache free" memory as
> reported by `free`.  Jconsole reports that the max heap limit is being
> respected.  This continues over a few weeks until their is so little
> page cache that performance degrades or the kernel oom killer steps in.
>
> The program (Apache Cassandra) is not configured to use mmap or as far
> as I can tell anything improper with direct allocation.   I understand
> that the JVM has internal structures that take memory, that GC can only
> periodically compact the heap, etc.  However, unless I am missing
> something the RES should eventually (say within days) come down.  This
> is the case with every other java program I have monitored, but again
> unless I'm missing something it should not be possible for any Java
> program to induce this different behaviour.
>
> Since bugs in the JVM or libc level seem unlikely I have been searching
> for a document along the lines of "you are probaly wrong, here are all
> the things you should do before claiming you found a memory leak in a
> JVM", but have been unable to find one.  The "Java Trouble-Shooting and
> Diagnostic Guide" contains section on OOM Exceptions and JIN, but not
> hotspot itself.  If someone could point me in the direction of one I
> would appreciate it.
>
> Short of that are there steps to make a useful bug report short of "run
> the program with valgrind for n days"?
>
> Thank you for your time,
> Chris Burroughs
>
> [1] http://img16.imageshack.us/img16/4813/cassandrarsssmooth.png