RFC: Unhoarding heap for bursty applications

[Aleksey Shipilev]

"Step right up, get a piece of your heap back, step right up!"

I get this question every so often: what would it take for Shenandoah to return memory to OS? It
seems. Today I hacked together a prototype with two ideas that might be worthwhile there.

It seems that the target are the applications that have bursty behaviors: e.g. those that become
inactive for a long time, and do not need to hoard the heap. The applications that allocate
continuously are not the target here: because under current policies, we want to use as much heap as
possible to make GC cycles less frequent, and releasing the heap would be futile, as we will claim
it back before the next cycle.

But for those bursty applications, there are two ideas:

 1. "Decayed uncommit". Naively, since our regions are page-aligned and page-sized, we can
selectively uncommit the regions when they are recycled. That would require committing them back as
we allocate new data in them, which raises performance questions for active applications. So, what
if we make the uncommit *decayed*, e.g. record the timestamp of last recycle, and if region was not
touched for a while, uncommit it. That would guarantee that actively used regions are never
uncommitted, but actually dormant ones are uncommitted, which is good.

 2. "Periodic GC". Our current GC trigger is mostly dependent on actual allocations (e.g. either the
allocation failure, or heap occupancy threshold is observed). So, if application abruptly becomes
idle, we would not do any GCs, which will keep regions from recycling, which will make uncommit not
that effective. Since our GC is almost fully concurrent, there seems to be no issues with doing GC
just because we feel like it.

Crude and dirty prototype:
 http://cr.openjdk.java.net/~shade/shenandoah/wip-unhoarding/webrev.01/

Performance experiment with a synthetic bursty workload:
 http://cr.openjdk.java.net/~shade/shenandoah/wip-unhoarding/run.sh
 http://cr.openjdk.java.net/~shade/shenandoah/wip-unhoarding/unhoard-1.png

With both uncommit and periodic GC, we drop RSS to ~30 MB, without any additional tuning. I think
this enough for the proof of concept. It seems both ideas are not hard to productize.

Comments? Objections? Better ideas?

Thanks,
-Aleksey