G1 discovers same garbage again?

Florian Binder java at java4.info
Wed Nov 30 00:35:45 PST 2011 

Hi Tony,

first of all thank you for your answere.
See inline.

Am 29.11.2011 18:29, schrieb Tony Printezis:
> Hi Florian,
>
> See inline.
>
> On 11/28/2011 2:18 PM, Florian Binder wrote:
>> Hi everybody,
>>
>> I have a java application with 20gb (large-table) memory and using the
>> g1 garbage collector.
>
> Quick clarification: I saw that you use a 20G heap from the parameters 
> you showed below. Do you know what's your live data size?
At this time I have about 14gb live data but it is growing day by day.
>
>> The application calculates the whole time with 10 threads some ratios
>> (high cpu load). This is done without producing any garbage. About two
>> times a minute a request is sent which produce a littlebit garbage.
>> Since we are working with realtime data we are interested in very short
>> stop-the-world pauses. Therefore we have used the CMS gc in the past
>> until we have got problems with fragmentation now.
>
> Since you don't produce much garbage how come you have fragmentation? 
> Do you keep the results for all the requests you serve?
This data is hold one day and every night it is droped and reinitialized.
We have a lot of different server with big memory and have had problems 
with fragmentation on few of them. This was the cause I am experiencing 
with g1 in general. I am not sure if we had fragmentation on this one.

Today I tried the g1 with another server which surely have had a problem 
with fragmented heap, but this one did not start wit g1. I got several 
different exceptions (NoClassDefFound, NullPointerException or even a 
jvm-crash ;-)). But I think I will write you another email especially 
for this, because it is started with a lot of special parameters (e.g. 
-Xms39G -Xmx39G -XX:+UseCompressedOops -XX:ObjectAlignmentInBytes=16 
-XX:+UseLargePages).

>
>> Therefore I am trying the g1.
>>
>> This seemed to work very well at first. The stw-pauses were, except the
>> cleanup pause,
>
> Out of curiosity: how long are the cleanup pauses?
I think they were about 150ms. This is acceptable for me, but in 
proportion to the garbage-collection of 30ms it is very long and 
therefore I was wondering.
>
>> very short. This yields me to my first question:
>> Is this normal and are there any parameters to influence the
>> cleanup-process?
>
> I don't think there's much you can do in the app to influence the 
> cleanup duration. During this pause we do some, ahem, cleanup of our 
> data structures and for large heaps I have also seen the cleanup 
> pauses to take longer than I thought they would take. I know this is 
> not going to help you in the short term but we have plans to do the 
> cleanup work concurrently (or at least mostly-concurrently) in the 
> future.
Sounds good :-)
>
>>   I thought this phase should be short because there is
>> just finished the counting, the role of the bitmaps is switched and the
>> next possible garbage regions are detemined. All things, which should be
>> very fast. So what is taking the time?
>
> Most likely, the remembered set scrubbing phase...
>
>> The second cause for my email is the crazy behaviour after a few hours:
>> After the startup of the server it uses about 13.5 gb old-gen memory and
>> generates very slowly eden-garbage. Since the new allocated memory is
>> mostly garbage the (young) garbage collections are very fast and g1
>> decides to grow up the eden space. This works 4 times until eden space
>> has more than about 3.5 gb memory. After this the gc is making much more
>> collections and while the collections it discovers new garbage (probably
>> the old one again).
>
> I'm not quite sure what you mean by "it discovers new garbage". For 
> young GCs, G1 (and our other GCs) will reclaim any young objects that 
> will discover to be dead (more accurately: that it will not discover 
> to be live).
>
>> Eden memory usage jumps between 0 and 3.5gb even
>> though I am sure the java-application is not making more than before.
>
> Well, that's not good. :-) Can you try to explicitly set the young gen 
> size with -Xmn3g say, to see what happens?
With "it discovers new garbage" I mean that during the garbage 
collection the eden space usage jumps up to 3gb. Then it cleans up the 
whole garbage (eden usage is 0) and a few seconds later the eden usage 
jumps again up. You can see this in the 1h eden-space snapshot:
http://java4.info/g1/eden_1h.png
Since the jumps are betweend 0 and the last max eden usage (of about 
3.5gb) I assume that it discovers the same garbage, it cleaned up the 
last time, and collects it again. I am sure the application is not 
making more garbage than the time before. Have you ever heared of 
problems like this?
After I have written the last email, I have seen that it has calm itself 
after a few hours. But it is nevertheless very curious and produces a 
lot of unnecessary pauses.

Flo

>
> Tony
>
>>   I
>> assume that it runs during a collection in the old garbage and collects
>> it again. Is this possible? Or is there an overflow since eden space
>> uses more than 3.5 gb?
>>
>> Thanks and regards,
>> Flo
>>
>> Some useful information:
>> $ java -version
>> java version "1.6.0_29"
>> Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
>>
>> Startup Parameters:
>> -Xms20g -Xmx20g
>> -verbose:gc \
>> -XX:+UnlockExperimentalVMOptions \
>> -XX:+UseG1GC \
>> -XX:+PrintGCDetails \
>> -XX:+PrintGCDateStamps \
>> -XX:+UseLargePages \
>> -XX:+PrintFlagsFinal \
>> -XX:-TraceClassUnloading \
>>
>> $ cat /proc/meminfo | grep Huge
>> HugePages_Total: 11264
>> HugePages_Free:   1015
>> HugePages_Rsvd:     32
>> Hugepagesize:     2048 kB
>>
>> A few screen-shots of the jconsole memory-view:
>> http://java4.info/g1/1h.png
>> http://java4.info/g1/all.png
>> http://java4.info/g1/eden_1h.png
>> http://java4.info/g1/eden_all.png
>> http://java4.info/g1/oldgen_all.png
>>
>> The sysout end syserr logfile with the gc logging and PrintFinalFlags:
>> http://java4.info/g1/out_err.log.gz
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

More information about the hotspot-gc-use mailing list