RFC / RFR: 8210473: JEP 345: NUMA-Aware Memory Allocation for G1

[sangheon.kim at oracle.com]

Hi all,

Could I have some comments / reviews for these implementations for 
making G1 NUMA aware?

1. Concept
The given Java Heap regions will be split and touched by memory touch 
threads to make the regions belong to active NUMA nodes. And then when 
mutators request memory, the mutators will get memory from local node. 
Not only mutators, but also survivors also will have same node memory 
during copying to survivor region state.
For more details, please refer to the JEP 345: JDK-8210473 
<https://bugs.openjdk.java.net/browse/JDK-8210473>.

FYI, if we enable -XX:+UseNUMA on current G1, OS will try to interleave 
memory, so we are already getting some benefit from it. I. e. G1 code is 
not NUMA aware but UseNUMAInterleaving is automatically enabled via 
UseNUMA.

2. Improvements
- I compared "legacy +UseNUMA (without the patch)" with "new +UseNUMA 
(with patch)" and I could see some improvements like below.
    a. Benchmark scores: max-JOPS on SpecJBB2015, score on SpecJBB2005 
around 1 ~ 6%. SpecJVM2008: 1 ~ 5%
    b. +UseNUMA shows better performance than -UseNUMA on above benchmarks.
- Current scope (NUMA aware memory allocation) gives limited 
improvements but I expect following patches will make it better.

3. Known limitations
- More GCs: current G1 is already wasting the last survivor region but 
the new implementation makes it worse as there are survivor region per 
NUMA node. JDK-8220089 is to address this problem and its working patch 
makes less frequent GC.
- No pause time improvement: this is expected as worker threads are 
still may process foreign node allocated objects. And this is not scope 
of this JEP.

4. Follow-up enhancements
-  JDK-8220089 G1 reuse survivor region as eden region at the end of gc: 
this enhancement minimizes wasting survivor region which the NUMA 
implementation makes this wasting situation worse. Initially I tried to 
push 8220089 before this JEP but there are still dependent / blocking 
issues at 8220089. Since all related CRs are targeting to JDK-14, I'm 
processing NUMA JEP first.
- JDK-8230411 NUMA aware work gang: current memory touch threads will be 
replaced with NUMA aware work gang.
- JDK-8230412 NUMA aware task processing: when gang workers do their 
works, threads may process foreign node allocated objects. This CR is to 
minimize this case.
- Make G1 code more and more NUMA aware.

For easier review, I separated into 3 pieces of patches. But I would 
like to push all together.
1/3: Mutator allocation
2/3: Survivor allocation
3/3: Stat / logging

Thanks,
Sangheon