G1 discovers same garbage again?

[Florian Binder]

Hi everybody,

I have a java application with 20gb (large-table) memory and using the 
g1 garbage collector.
The application calculates the whole time with 10 threads some ratios 
(high cpu load). This is done without producing any garbage. About two 
times a minute a request is sent which produce a littlebit garbage. 
Since we are working with realtime data we are interested in very short 
stop-the-world pauses. Therefore we have used the CMS gc in the past 
until we have got problems with fragmentation now.
Therefore I am trying the g1.

This seemed to work very well at first. The stw-pauses were, except the 
cleanup pause, very short. This yields me to my first question:
Is this normal and are there any parameters to influence the 
cleanup-process? I thought this phase should be short because there is 
just finished the counting, the role of the bitmaps is switched and the 
next possible garbage regions are detemined. All things, which should be 
very fast. So what is taking the time?

The second cause for my email is the crazy behaviour after a few hours:
After the startup of the server it uses about 13.5 gb old-gen memory and 
generates very slowly eden-garbage. Since the new allocated memory is 
mostly garbage the (young) garbage collections are very fast and g1 
decides to grow up the eden space. This works 4 times until eden space 
has more than about 3.5 gb memory. After this the gc is making much more 
collections and while the collections it discovers new garbage (probably 
the old one again). Eden memory usage jumps between 0 and 3.5gb even 
though I am sure the java-application is not making more than before. I 
assume that it runs during a collection in the old garbage and collects 
it again. Is this possible? Or is there an overflow since eden space 
uses more than 3.5 gb?

Thanks and regards,
Flo

Some useful information:
$ java -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)

Startup Parameters:
-Xms20g -Xmx20g
-verbose:gc \
-XX:+UnlockExperimentalVMOptions \
-XX:+UseG1GC \
-XX:+PrintGCDetails \
-XX:+PrintGCDateStamps \
-XX:+UseLargePages \
-XX:+PrintFlagsFinal \
-XX:-TraceClassUnloading \

$ cat /proc/meminfo | grep Huge
HugePages_Total: 11264
HugePages_Free:   1015
HugePages_Rsvd:     32
Hugepagesize:     2048 kB

A few screen-shots of the jconsole memory-view:
http://java4.info/g1/1h.png
http://java4.info/g1/all.png
http://java4.info/g1/eden_1h.png
http://java4.info/g1/eden_all.png
http://java4.info/g1/oldgen_all.png

The sysout end syserr logfile with the gc logging and PrintFinalFlags:
http://java4.info/g1/out_err.log.gz