[11] RFR(M): 8189922: UseNUMA memory interleaving vs membind

[Swati Sharma]

Hi All,

Here is the numa information of the system :
swati at java-diesel1:~$ numactl -H
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 4 5 6 7 64 65 66 67 68 69 70 71
node 0 size: 64386 MB
node 0 free: 64134 MB
node 1 cpus: 8 9 10 11 12 13 14 15 72 73 74 75 76 77 78 79
node 1 size: 64509 MB
node 1 free: 64232 MB
node 2 cpus: 16 17 18 19 20 21 22 23 80 81 82 83 84 85 86 87
node 2 size: 64509 MB
node 2 free: 64215 MB
node 3 cpus: 24 25 26 27 28 29 30 31 88 89 90 91 92 93 94 95
node 3 size: 64509 MB
node 3 free: 64157 MB
node 4 cpus: 32 33 34 35 36 37 38 39 96 97 98 99 100 101 102 103
node 4 size: 64509 MB
node 4 free: 64336 MB
node 5 cpus: 40 41 42 43 44 45 46 47 104 105 106 107 108 109 110 111
node 5 size: 64509 MB
node 5 free: 64352 MB
node 6 cpus: 48 49 50 51 52 53 54 55 112 113 114 115 116 117 118 119
node 6 size: 64509 MB
node 6 free: 64359 MB
node 7 cpus: 56 57 58 59 60 61 62 63 120 121 122 123 124 125 126 127
node 7 size: 64508 MB
node 7 free: 64350 MB
node distances:
node   0   1   2   3   4   5   6   7
  0:  10  16  16  16  32  32  32  32
  1:  16  10  16  16  32  32  32  32
  2:  16  16  10  16  32  32  32  32
  3:  16  16  16  10  32  32  32  32
  4:  32  32  32  32  10  16  16  16
  5:  32  32  32  32  16  10  16  16
  6:  32  32  32  32  16  16  10  16
  7:  32  32  32  32  16  16  16  10

Thanks,
Swati

On Tue, Jun 19, 2018 at 12:00 AM, Gustavo Romero <gromero at linux.vnet.ibm.com
> wrote:

> Hi Swati,
>
> On 06/16/2018 02:52 PM, Swati Sharma wrote:
>
>> Hi All,
>>
>> This is my first patch,I would appreciate if anyone can review the fix:
>>
>> Bug : https://bugs.openjdk.java.net/browse/JDK-8189922 <
>> https://bugs.openjdk.java.net/browse/JDK-8189922>
>> Webrev :http://cr.openjdk.java.net/~gromero/8189922/v1
>>
>> The bug is about JVM flag UseNUMA which bypasses the user specified
>> numactl --membind option and divides the whole heap in lgrps according to
>> available numa nodes.
>>
>> The proposed solution is to disable UseNUMA if bound to single numa node.
>> In case more than one numa node binding, create the lgrps according to
>> bound nodes.If there is no binding, then JVM will divide the whole heap
>> based on the number of NUMA nodes available on the system.
>>
>> I appreciate Gustavo's help for fixing the thread allocation based on
>> numa distance for membind which was a dangling issue associated with main
>> patch.
>>
>
> Thanks. I have no further comments on it. LGTM.
>
>
> Best regards,
> Gustavo
>
> PS: Please, provide numactl -H information when possible. It helps to grasp
> promptly the actual NUMA topology in question :)
>
> Tested the fix by running specjbb2015 composite workload on 8 NUMA node
>> system.
>> Case 1 : Single NUMA node bind
>> numactl --cpunodebind=0 --membind=0 java -Xmx24g -Xms24g -Xmn22g
>> -XX:+UseNUMA -Xlog:gc*=debug:file=gc.log:time,uptimemillis
>> <composite_application>
>> Before Patch: gc.log
>> eden space 22511616K(22GB), 12% used
>>      lgrp 0 space 2813952K, 100% used
>>      lgrp 1 space 2813952K, 0% used
>>      lgrp 2 space 2813952K, 0% used
>>      lgrp 3 space 2813952K, 0% used
>>      lgrp 4 space 2813952K, 0% used
>>      lgrp 5 space 2813952K, 0% used
>>      lgrp 6 space 2813952K, 0% used
>>      lgrp 7 space 2813952K, 0% used
>> After Patch : gc.log
>> eden space 46718976K(45GB), 99% used(NUMA disabled)
>>
>> Case 2 : Multiple NUMA node bind
>> numactl --cpunodebind=0,7 –membind=0,7 java -Xms50g -Xmx50g -Xmn45g
>> -XX:+UseNUMA -Xlog:gc*=debug:file=gc.log:time,uptimemillis
>> <composite_application>
>> Before Patch :gc.log
>> eden space 46718976K, 6% used
>>      lgrp 0 space 5838848K, 14% used
>>      lgrp 1 space 5838848K, 0% used
>>      lgrp 2 space 5838848K, 0% used
>>      lgrp 3 space 5838848K, 0% used
>>      lgrp 4 space 5838848K, 0% used
>>      lgrp 5 space 5838848K, 0% used
>>      lgrp 6 space 5838848K, 0% used
>>      lgrp 7 space 5847040K, 35% used
>> After Patch : gc.log
>> eden space 46718976K(45GB), 99% used
>>       lgrp 0 space 23359488K(23.5GB), 100% used
>>       lgrp 7 space 23359488K(23.5GB), 99% used
>>
>>
>> Note: The proposed solution is only for numactl membind option.The fix is
>> not for --cpunodebind and localalloc which is a separate bug bug
>> https://bugs.openjdk.java.net/browse/JDK-8205051 and fix is in progress
>> on this.
>>
>> Thanks,
>> Swati Sharma
>> Software Engineer -2 at AMD
>>
>>
>