AW: Why G1 doesn't cut it for our application
Cornelius Riemenschneider
cri at itscope.de
Thu Apr 10 11:17:07 UTC 2014
Hey,
first, we run two different types of servers ? one of them is very allocation heavy, with the possibility of huge objects whcih live longer.
The other one usually has only small allocations, but some requests may trigger the allocation of up to ~150MB of data at once.
This data is used at whole, but very short-lived. Explanation:
We sometimes ask the question: Given two unsorted sets of integer IDs, how many common ids are in both?
Aka the cardinal number of the intersection of the two sets.
The fastest way we found to determine is this:
for both sets
determine the biggest integer per set (linear scan of the array)
allocate a long array which has as many bits as the alrgest id requires
for each id, set the bit which corresponds with the id to 1 (example: id 3 in ou set would mean the first long in our long array, and then the 3rd bit there)
now, we can intersect these sets very fast by bitwise and of each long in the array and using Long.popcnt, which is documented to compile to the POPCNT instruction on our CPUs.
The intersect-and-count is very fast (I don?t have exact numbers on this, but it should be <300ms), but the objects required are rather huge.
I even found this gem in our log:
485471.317: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: humongous allocation request failed, allocation request: 81000016 bytes]
485471.317: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 67108864 bytes, attempted expansion amount: 67108864 bytes]
485471.317: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: heap expansion operation failed]
485471.319: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: humongous allocation request failed, allocation request: 81000016 bytes]
485471.319: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 67108864 bytes, attempted expansion amount: 67108864 bytes]
485471.319: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: heap expansion operation failed]
485471.319: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: allocation request failed, allocation request: 81000016 bytes]
485471.319: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 81000016 bytes, attempted expansion amount: 83886080 bytes]
485471.319: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: heap expansion operation failed]
2014-04-10T11:58:26.028+0200: 485471.319: [Full GC 11G->7884M(20G), 20.8011100 secs]
[Eden: 0.0B(1584.0M)->0.0B(1600.0M) Survivors: 8192.0K->0.0B Heap: 11.3G(20.0G)->7884.1M(20.0G)]
[Times: user=37.88 sys=0.00, real=20.81 secs]
about 80MB allocation, 9GB free but it still requires a fullgc (though still with 7u51).
The settings used there are:
-Xms20g
-Xmx20g
-XX:ReservedCodeCacheSize=256m
-XX:PermSize=256m
-XX:MaxPermSize=768m
-XX:+ExplicitGCInvokesConcurrent
-XX:+PrintAdaptiveSizePolicy
-XX:+UseG1GC
-XX:MaxGCPauseMillis=80
-XX:InitiatingHeapOccupancyPercent=45
-XX:G1ReservePercent=15
A CMS config which has way less problems for the server would be:
-Xms20g
-Xmx20g
-XX:ReservedCodeCacheSize=256m
-XX:PermSize=256m
-XX:MaxPermSize=768m
-XX:NewRatio=3
-XX:SurvivorRatio=5
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=75
-XX:+CMSClassUnloadingEnabled
-XX:+ExplicitGCInvokesConcurrent
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+CMSScavengeBeforeRemark
I can?t increase the heap further without deploying more RAM.
I tested a lot more different options with out allocation heavy servers, because restarting these has no direct impact on our customers.
The configuration which worked best used the following: A high pause time goal (600ms), but a limit to the newsize (6g), so that we still have enough oldspace for the big objects.
I also tried setting the regionsize to 32mb, but this didn?t cause a noticable difference from the default. Setting InitiatingHeapSize to 0, 1 or 45 didn?t impact the amount of fullgcs either (at least not noticable).
When i?ll test with 8u20, I?ll run with +PrintHeapAtGCExtended.
Regards,
Cornelius Riemenschneider
--
ITscope GmbH
Ludwig-Erhard-Allee 20
76131 Karlsruhe
Email: cornelius.riemenschneider at itscope.de
https://www.itscope.com
Handelsregister: AG Mannheim, HRB 232782
Sitz der Gesellschaft: Karlsruhe
Geschäftsführer: Alexander Münkel, Benjamin Mund, Stefan Reger
Von: Srinivas Ramakrishna [mailto:ysr1729 at gmail.com]
Gesendet: Mittwoch, 9. April 2014 18:50
An: Cornelius Riemenschneider
Cc: hotspot-gc-use at openjdk.java.net
Betreff: Re: Why G1 doesn't cut it for our application
May be look at the +PrintHeapAtGCExtended (or similar) option which gives you a breakdown of space usage in each of the regions.
The messages here state that the attempt is to increase the heap size by a mere 117 MB. It would have been nice to also print the
size of the allocation request that was failing and causing a fallback to full gc. It seems a bit strange that with only 11 G of the 17 G
in the old gen used (i.e. 6 GB free), the space was so fragmented as to prevent, presumably here, a 117 MB allocation (OTOH perhaps
the heap expansion request of 117 MB should not be construed as a request for 117 MB). Knowing the distribution of free space in
the regions before the full gc and the size of the allocation request that failed might provide some clues on how G1 may have to be
"paced" or tuned to keep sufficiently many completely free (and hopefully contiguous) regions to accomodate larger than region size
object allocations.
What region size are you using? I would definitely try a large region size and a larger heap, if you have sufficient free ram
(make the space available to the old generation three times the footprint of your application just to give G1 sufficient head
room to manage the space). I wouldn't necessarily want to force large object allocations into Eden unless you know that these objects
have a relatively short lifetime (which would seem to be a dubious use of large objects since it's likely then that a very small portion
of that object is actually used in that case).
What settings do you use with CMS? And what settings are you using with G1?
USD 0.02.
-- ramki
On Wed, Apr 9, 2014 at 4:56 AM, Cornelius Riemenschneider <cri at itscope.de> wrote:
Hello,
after recently switching to the latest java 7 (u51), I was eager to try out G1.
I used mainly http://www.slideshare.net/MonicaBeckwith/garbage-first-garbage-collector-g1-gc-migration-to-expectations-and-advanced-tuning for tuning, but
I hit a roadblock which makes it impossile for us to use G1.
Our allocation pattern includes sometimes huge objects, sometimes in the range of ~120MB, sometimes ~600MB, but I?ve seen about 1.2GB as well. This is obviously unfriendly to the GC.
Our tuned CMS mostly handles this, but sometimes we hit problems, so we had high expectations for G1.
G1, in our case, triggers FullGC way more often than CMS, even when the heap is mostly empty.
An example log excerpt for this:
399934.892: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: humongous allocation request failed, allocation request: 117245600 bytes]
399934.892: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 83886080 bytes, attempted expansion amount: 83886080 bytes]
399934.892: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: heap expansion operation failed]
399934.893: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: humongous allocation request failed, allocation request: 117245600 bytes]
399934.893: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 83886080 bytes, attempted expansion amount: 83886080 bytes]
399934.893: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: heap expansion operation failed]
399934.893: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: allocation request failed, allocation request: 117245600 bytes]
399934.893: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 117245600 bytes, attempted expansion amount: 117440512 bytes]
399934.893: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: heap expansion operation failed]
2014-04-09T12:12:49.602+0200: 399934.894: [Full GC 11G->8118M(20G), 20.8752850 secs]
[Eden: 8192.0K(1016.0M)->0.0B(2728.0M) Survivors: 96.0M->0.0B Heap: 11.6G(20.0G)->8118.8M(20.0G)]
[Times: user=37.77 sys=0.00, real=20.88 secs]
2014-04-09T12:13:10.479+0200: 399955.770: [GC concurrent-mark-abort]
We have a total of 20G for the heap available, and try to allocate objects in the 120MB range.
9 GB of the heap are free, so these should fit in without problems, even in Eden is a lot of free space.
The attempted heap expansion fails, because we use
-Xms20g
-Xmx20g
which is the maximum the server is able to handle.
Still, G1 gets us a FullGC here. This FullGC may be faster than a CMS FullGC, but these happen way too often to be tolerated, especially as this server is responsible for a web
application with which users directly interact ? 20 secs pause after clicking are simply not tolerable.
Besides using CMS, or not doing large allocations (which is sometimes impossible, given that we deal with a lot of data),
do you have oher ideas?
Is it known that an allocation pattern with a lot of huge objects breaks G1?
The above linked presentation suggests to increase the G1 region size when humongous allocation requests are encountered, so these allocation go in eden, but we can
not increase the region size beyond 32M, so this fix doesn?t work for us.
Regards,
Cornelius Riemenschneider
--
ITscope GmbH
Ludwig-Erhard-Allee 20
76131 Karlsruhe
Email: cornelius.riemenschneider at itscope.de
https://www.itscope.com
Handelsregister: AG Mannheim, HRB 232782
Sitz der Gesellschaft: Karlsruhe
Geschäftsführer: Alexander Münkel, Benjamin Mund, Stefan Reger
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20140410/6b92ebeb/attachment-0001.html>
More information about the hotspot-gc-use
mailing list