Reserved space in GC/TLABs

[Aleksey Shipilev]

Hi,

So I've been chasing a weird bimodal behavior in our mark tests. The test
retains the large immutable tree in the heap, and allocates objects around it.
The intent is that GC would clean up new allocations, after marking the
immutable tree.

However, with "aggressive" heuristics, there are two clear phases in workload life:
  a) Choosing the regions with the immutable tree, and evacuating it. This takes
a long time for a large tree;
  b) Choosing a few (two, basically) regions to promote the current chunk of new
roots -- probably the objects that happened to be temporarily live at safepoint.

The difference in performance between (a) and (b) are drastic. Looking closely
at (a), we can notice that otherwise immutable regions are chosen because there
used-live = 576 words:

[1.205s][info][gc] Choose region 33 with garbage = 576 and live = 4193728

"aggressive" chooses any region with garbage > 0.

After digging around, I realized those 576 words are the TLAB reserved space!
(Our promotion goes through GCLAB, which is technically the same thing). It goes
down once I tune -XX:AllocatePrefetchLines and friends. The comment in
ThreadLocalAllocBuffer::startup_initialization() mentions this reserved space is
needed to avoid faulting for going beyond the heap. (Why this is not in GC
storage management code, but handled for each TLAB, is beyond me at this point).

But perhaps a more interesting question is why phase (b) is immune to this. The
short answer is because mark-compact _ignores_ all those reserved space things,
because it does not deal with GCLABs at all.

So, there are few issues:

 1) GC/TLAB reserved space means every region has "garbage", which has
heuristics implications. We can check for "garbage > TLAB::alignment_reserve()"
in "aggressive"?

 2) More concerning: after mark-compact we can read beyound the heap, if we are
alloc-prefetching at the last region, near its end. We probably want to reserve
a region at the end of the heap to handle this?

Thanks,
-Aleksey