Full GC run freeing no memory despite many unreachable objects

[Darragh Curran]

Hi,

I hope someone can help me understand this better.

I'm running java version "Java(TM) SE Runtime Environment (build
1.6.0_19-b04) Java HotSpot(TM) Server VM (build 16.2-b04, mixed mode)"

It runs tomcat, with jvm heap options '-Xmx1600m -Xms1600m'

Every few weeks a host becomes unavailable for requests. When I look
into it, the java process is using 100% CPU, seems to have no running
threads when I do multiple stack dumps, and based on jstat output
appears to spend all it's time doing full gc runs:

14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1107  5656.903 6945.644
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1108  5666.547 6955.287
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1108  5666.547 6955.287
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1108  5666.547 6955.287
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1108  5666.547 6955.287
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1108  5666.547 6955.287
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1108  5666.547 6955.287
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1108  5666.547 6955.287
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1108  5666.547 6955.287
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1108  5666.547 6955.287
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1109  5675.945 6964.686
14400.0 13824.0  0.0    0.0   517888.0 517888.0 1092288.0  1060014.1
37376.0 37182.0  38654 1288.741 1109  5675.945 6964.686

I used jmap to dump the heap for analysis. When I analyse it I see
that 94% of objects are unreachable, but not yet collected.

>From jstat it appears that gc is running roughly every 10 seconds and
lasting approx 10 seconds, but fails to free any memory.

I'd really appreciate some advise on how to better understand my
problem and what to do to try and fix it.

Thanks,
Darragh