[11] RFR(M): 8189922: UseNUMA memory interleaving vs membind

[Swati Sharma]

 Hi All,

This is my first patch,I would appreciate if anyone can review the fix:

Bug : https://bugs.openjdk.java.net/browse/JDK-8189922
Webrev : http://cr.openjdk.java.net/~gromero/8189922/v1

The bug is about JVM flag UseNUMA which bypasses the user specified numactl
--membind option and divides the whole heap in lgrps according to available
numa nodes.

The proposed solution is to disable UseNUMA if bound to single numa node.
In case more than one numa node binding, create the lgrps according to
bound nodes.If there is no binding, then JVM will divide the whole heap
based on the number of NUMA nodes available on the system.

I appreciate Gustavo's help for fixing the thread allocation based on numa
distance for membind which was a dangling issue associated with main patch.

Tested the fix by running specjbb2015 composite workload on 8 NUMA node
system.
Case 1 : Single NUMA node bind
numactl --cpunodebind=0 --membind=0 java -Xmx24g -Xms24g -Xmn22g
-XX:+UseNUMA  -Xlog:gc*=debug:file=gc.log:time,uptimemillis
<composite_application>
Before Patch: gc.log
  eden space 22511616K(22GB), 12% used
    lgrp 0 space 2813952K, 100% used
    lgrp 1 space 2813952K, 0% used
    lgrp 2 space 2813952K, 0% used
    lgrp 3 space 2813952K, 0% used
    lgrp 4 space 2813952K, 0% used
    lgrp 5 space 2813952K, 0% used
    lgrp 6 space 2813952K, 0% used
    lgrp 7 space 2813952K, 0% used
After Patch : gc.log
    eden space 46718976K(45GB), 99% used (NUMA disabled)

Case 2 : Multiple NUMA node bind
numactl --cpunodebind=0,7 –membind=0,7 java -Xms50g -Xmx50g -Xmn45g
 -XX:+UseNUMA  -Xlog:gc*=debug:file=gc.log:time,uptimemillis
<composite_application>
Before Patch :gc.log
eden space 46718976K, 6% used
    lgrp 0 space 5838848K, 14% used
    lgrp 1 space 5838848K, 0% used
    lgrp 2 space 5838848K, 0% used
    lgrp 3 space 5838848K, 0% used
    lgrp 4 space 5838848K, 0% used
    lgrp 5 space 5838848K, 0% used
    lgrp 6 space 5838848K, 0% used
    lgrp 7 space 5847040K, 35% used
After Patch : gc.log
  eden space 46718976K(45GB), 99% used
     lgrp 0 space 23359488K(23.5GB), 100% used
     lgrp 7 space 23359488K(23.5GB), 99% used


Note: The proposed solution is only for numactl membind option.The fix is
not for --cpunodebind and localalloc which is a separate bug bug
https://bugs.openjdk.java.net/browse/JDK-8205051 and fix is in progress on
this.

Thanks,
Swati Sharma
Software Engineer -2 at AMD