Debugging Resident Set Size growth

[Y. Srinivas Ramakrishna]

On 4/13/2011 6:55 PM, Y. S. Ramakrishna wrote:
> Can you use pmap -x to see where and what the growth is in?

BTW, did you ever happen to get the pmap's to see where the growth
was happening?

-- ramki


> For example, if your Java heap needs increase then the
> committed part of the heap will become resident as and
> when those pages are actually touched or used. So
> with a program that eventually grows to occupy the heap
> it's been given, one would eventually expect to reach 1.5 G
> and some more (for the perm gen, the C-heap and so forth).
> I am assuming that the unit on the y-axis of your graph is
> KB, so that at the right end of your graph you are
> reaching an RSS of a bit over 2 G. (I also assume from
> you description that your x-axis is about 72 hours, although
> there are no labels.)
>
> Did you look at the GC logs or at the Java heap with jmap or jconsole?
>
> In summary, without more detaiol, I do not see that there is
> necessarily a problem here to solve (although i do sympathise
> with the Linux OOM killer, I am sure it will not step in
> just because your RSS is 2 GB for a program that you seek
> to give 1.5 GB of java heap; how much RAM does your machine
> have?)
>
> -- ramki
>
> On 04/13/11 18:30, Chris Burroughs wrote:
>> Hello all. I am experiencing a problem with a java program on running
>> 1.6.0_23 on RHEL5. The resident set size (as reported by
>> /proc/$PID/status for example) of the process continues to grow. For
>> example here [1] is the RES of the process from approximately the 24th
>> to 72nd hours of operation with min and max heap equal to 1.5 GB. This
>> corresponds to a an decrease in "-/+ buffers/cache free" memory as
>> reported by `free`. Jconsole reports that the max heap limit is being
>> respected. This continues over a few weeks until their is so little
>> page cache that performance degrades or the kernel oom killer steps in.
>>
>> The program (Apache Cassandra) is not configured to use mmap or as far
>> as I can tell anything improper with direct allocation. I understand
>> that the JVM has internal structures that take memory, that GC can only
>> periodically compact the heap, etc. However, unless I am missing
>> something the RES should eventually (say within days) come down. This
>> is the case with every other java program I have monitored, but again
>> unless I'm missing something it should not be possible for any Java
>> program to induce this different behaviour.
>>
>> Since bugs in the JVM or libc level seem unlikely I have been searching
>> for a document along the lines of "you are probaly wrong, here are all
>> the things you should do before claiming you found a memory leak in a
>> JVM", but have been unable to find one. The "Java Trouble-Shooting and
>> Diagnostic Guide" contains section on OOM Exceptions and JIN, but not
>> hotspot itself. If someone could point me in the direction of one I
>> would appreciate it.
>>
>> Short of that are there steps to make a useful bug report short of "run
>> the program with valgrind for n days"?
>>
>> Thank you for your time,
>> Chris Burroughs
>>
>> [1] http://img16.imageshack.us/img16/4813/cassandrarsssmooth.png