AW: how to tune gc for tomcat server on large machine that usesalmost all old generation smallish objects

[Bernd Eckenfels]

Dont worry about the sweep, G1 is like CMS mostly concurrent.

I would suggest to test it with GC log enabled and then you can worry. Most likely you want to allow it to kick of GC later so you can save some concurrent CPU. You also need to fear the FullGC when your regions become to fragmented (this hopefully does not happen if the LRU frees lots of object allocated at the same time in the same Region, but you never know. You might unfortunatelly Need to have 30% or more unused heap to defend against that.

There is BTW a Mailing list for GC Usage as opposed to development.

Gruss
Bernd
-- 
http://bernd.eckenfels.net

Von: Andy Nuss
Gesendet: Mittwoch, 13. Dezember 2017 22:56
An: Kirk Pepperdine; hotspot-gc-dev at openjdk.java.net
Betreff: Re: how to tune gc for tomcat server on large machine that usesalmost all old generation smallish objects

Let me try to explain.  On a 16 gig heap, I anticipate almost 97% of the heap in use at any given moment is ~30 and ~100 char strings.  The rest is small pointer objects in the ConcurrentHashMap, also longly held, and tomcat's nio stuff.  So at any moment in time, most of the in-use heap (and I will keep about 20% unused to aid gc), is a huge number of longly held strings.  Over time, as the single servlet receives requests to cache newly accessed key/val pairs, the number of strings grows to its maximum I allow.  At that point, a background thread sweeps away half of the LRU key/value pairs (30,100 char strings).  Now they are unreferenced and sweepable.  That's all I do.  Then the servlet keeps receiving requests to put more key/val pairs.  As well as handle get requests.  At the point in time where I clear all the LRU pairs, which might take minutes to iterate, G1 can start doing its thing, not that it will know to do so immediately.  I'm worried that whenever G1 does its thing, because the sweepable stuff is 100% small oldgen objects, servlet threads will timeout on the client side.  Not that this happens several times a day, but if G1 does take a long time to sweep a massive heap with all oldgen objects that are small, the *only* concern is that servlet requests will time out during this period.

Realize I know nothing about GC, except that periodically, eclipse hangs due to gc and then crashes on me.  I.e. after 4 hours of editing.  And that all the blogs I found talked about newgen and TLAB and other things assuming typical ephemeral usage going on which is not at all the case on this particular machine instance.  Again, all longly held small strings, growing and growing over time steadily, suddenly half are freed reference wise by me.

If there are no GC settings that make that sweepable stuff happen in a non-blocking thread, and tomcat's servlets could all hang once every other day for many many seconds on this 16 gig machine (the so-called long gc-pause that people blog about), that might motivate me to abandon this and use the memcached product.


On Wednesday, December 13, 2017, 12:15:38 PM PST, Kirk Pepperdine <kirk at kodewerk.com> wrote: 


Hi Andy,

On Dec 13, 2017, at 8:34 PM, Andy Nuss <andrew_nuss at yahoo.com> wrote:

Thanks Kirk,

The array is just a temporary buffer held onto that has its entries cleared to null after my LRU sweep.  The references that are freed to GC are in the ConcurrentHashMaps, and are all 30 char and 100 char strings, key/vals, but not precisely, so I assume that when I do my LRU sweep when needed, its freeing a ton of small strings, 



which G1 has to reallocate into bigger chunks, and mark freed, and so,

Not sure I understand this bit. Can you explain what you mean by this?


so that I can in the future add new such strings to the LRU cache.  The concern was whether this sweep of old gen strings scattered all over the huge heap would cause tomcat nio-based threads to "hang", not respond quickly, or would G1 do things less pre-emptively.  Are you basically saying that, "no tomcat servlet response time won't be significantly affected by G1 sweep”?

I’m not sure what you’re goal is here. I would say, design as needed and let the collector do it’s thing. That said, temporary humongous allocations are not well managed by the G1. Better to create up front and cache it for future downstream use.

As for a sweep… what I think you’re asking about is object copy costs. These costs should and typically do dominate pause time. Object copy cost is proportional to the number of live objects in the collection set (CSet). Strings are dedup’ed after age 5 so with most heap configurations, duplicate Strings will be dedup’ed before they hit tenured.


Also, I was wondering does anyone know how memcached works, and why it is used in preference to a custom design such as mine which seems a lot simpler.  I.e. it seems that with "memcached", you have to worry about "slabs" and memcached's own heap management, and waste a lot of memory.

I’m the wrong person to defend the use of memcached. It certainly does serve a purpose.. that said, to use it to offload temp object means you end up creating your own garbage collector… and as you can see by the efforts GC engineers put into each implementation, it’s a non-trivial under-taking.

Kind regards,
Kirk


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20171213/01af9a77/attachment.htm>