[ping] Re: [11] RFR(M): 8189922: UseNUMA memory interleaving vs membind

Fri Jun 29 16:46:20 UTC 2018

 Hi,

Could I get a review for this change that affects the JVM when there are
pinned memory nodes please?

It's already reviewed and tested on PPC64 and on AARCH64 by Gustavo and
Derek, however both are not Reviewers so I need additional reviews for that
change.

Thanks in advance.

Swati

On Tue, Jun 19, 2018 at 5:58 PM, Swati Sharma <swatibits14 at gmail.com> wrote:

> Hi All,
>
> Here is the numa information of the system :
> swati at java-diesel1:~$ numactl -H
> available: 8 nodes (0-7)
> node 0 cpus: 0 1 2 3 4 5 6 7 64 65 66 67 68 69 70 71
> node 0 size: 64386 MB
> node 0 free: 64134 MB
> node 1 cpus: 8 9 10 11 12 13 14 15 72 73 74 75 76 77 78 79
> node 1 size: 64509 MB
> node 1 free: 64232 MB
> node 2 cpus: 16 17 18 19 20 21 22 23 80 81 82 83 84 85 86 87
> node 2 size: 64509 MB
> node 2 free: 64215 MB
> node 3 cpus: 24 25 26 27 28 29 30 31 88 89 90 91 92 93 94 95
> node 3 size: 64509 MB
> node 3 free: 64157 MB
> node 4 cpus: 32 33 34 35 36 37 38 39 96 97 98 99 100 101 102 103
> node 4 size: 64509 MB
> node 4 free: 64336 MB
> node 5 cpus: 40 41 42 43 44 45 46 47 104 105 106 107 108 109 110 111
> node 5 size: 64509 MB
> node 5 free: 64352 MB
> node 6 cpus: 48 49 50 51 52 53 54 55 112 113 114 115 116 117 118 119
> node 6 size: 64509 MB
> node 6 free: 64359 MB
> node 7 cpus: 56 57 58 59 60 61 62 63 120 121 122 123 124 125 126 127
> node 7 size: 64508 MB
> node 7 free: 64350 MB
> node distances:
> node   0   1   2   3   4   5   6   7
>   0:  10  16  16  16  32  32  32  32
>   1:  16  10  16  16  32  32  32  32
>   2:  16  16  10  16  32  32  32  32
>   3:  16  16  16  10  32  32  32  32
>   4:  32  32  32  32  10  16  16  16
>   5:  32  32  32  32  16  10  16  16
>   6:  32  32  32  32  16  16  10  16
>   7:  32  32  32  32  16  16  16  10
>
> Thanks,
> Swati
>
> On Tue, Jun 19, 2018 at 12:00 AM, Gustavo Romero <
> gromero at linux.vnet.ibm.com> wrote:
>
>> Hi Swati,
>>
>> On 06/16/2018 02:52 PM, Swati Sharma wrote:
>>
>>> Hi All,
>>>
>>> This is my first patch,I would appreciate if anyone can review the fix:
>>>
>>> Bug : https://bugs.openjdk.java.net/browse/JDK-8189922 <
>>> https://bugs.openjdk.java.net/browse/JDK-8189922>
>>> Webrev :http://cr.openjdk.java.net/~gromero/8189922/v1
>>>
>>> The bug is about JVM flag UseNUMA which bypasses the user specified
>>> numactl --membind option and divides the whole heap in lgrps according to
>>> available numa nodes.
>>>
>>> The proposed solution is to disable UseNUMA if bound to single numa
>>> node. In case more than one numa node binding, create the lgrps according
>>> to bound nodes.If there is no binding, then JVM will divide the whole heap
>>> based on the number of NUMA nodes available on the system.
>>>
>>> I appreciate Gustavo's help for fixing the thread allocation based on
>>> numa distance for membind which was a dangling issue associated with main
>>> patch.
>>>
>>
>> Thanks. I have no further comments on it. LGTM.
>>
>>
>> Best regards,
>> Gustavo
>>
>> PS: Please, provide numactl -H information when possible. It helps to
>> grasp
>> promptly the actual NUMA topology in question :)
>>
>> Tested the fix by running specjbb2015 composite workload on 8 NUMA node
>>> system.
>>> Case 1 : Single NUMA node bind
>>> numactl --cpunodebind=0 --membind=0 java -Xmx24g -Xms24g -Xmn22g
>>> -XX:+UseNUMA -Xlog:gc*=debug:file=gc.log:time,uptimemillis
>>> <composite_application>
>>> Before Patch: gc.log
>>> eden space 22511616K(22GB), 12% used
>>>      lgrp 0 space 2813952K, 100% used
>>>      lgrp 1 space 2813952K, 0% used
>>>      lgrp 2 space 2813952K, 0% used
>>>      lgrp 3 space 2813952K, 0% used
>>>      lgrp 4 space 2813952K, 0% used
>>>      lgrp 5 space 2813952K, 0% used
>>>      lgrp 6 space 2813952K, 0% used
>>>      lgrp 7 space 2813952K, 0% used
>>> After Patch : gc.log
>>> eden space 46718976K(45GB), 99% used(NUMA disabled)
>>>
>>> Case 2 : Multiple NUMA node bind
>>> numactl --cpunodebind=0,7 –membind=0,7 java -Xms50g -Xmx50g -Xmn45g
>>> -XX:+UseNUMA -Xlog:gc*=debug:file=gc.log:time,uptimemillis
>>> <composite_application>
>>> Before Patch :gc.log
>>> eden space 46718976K, 6% used
>>>      lgrp 0 space 5838848K, 14% used
>>>      lgrp 1 space 5838848K, 0% used
>>>      lgrp 2 space 5838848K, 0% used
>>>      lgrp 3 space 5838848K, 0% used
>>>      lgrp 4 space 5838848K, 0% used
>>>      lgrp 5 space 5838848K, 0% used
>>>      lgrp 6 space 5838848K, 0% used
>>>      lgrp 7 space 5847040K, 35% used
>>> After Patch : gc.log
>>> eden space 46718976K(45GB), 99% used
>>>       lgrp 0 space 23359488K(23.5GB), 100% used
>>>       lgrp 7 space 23359488K(23.5GB), 99% used
>>>
>>>
>>> Note: The proposed solution is only for numactl membind option.The fix
>>> is not for --cpunodebind and localalloc which is a separate bug bug
>>> https://bugs.openjdk.java.net/browse/JDK-8205051 and fix is in progress
>>> on this.
>>>
>>> Thanks,
>>> Swati Sharma
>>> Software Engineer -2 at AMD
>>>
>>>
>>
>