UseNUMA membind Issue in openJDK

[White, Derek]

See inline:

> -----Original Message-----
> From: Gustavo Romero [mailto:gromero at linux.vnet.ibm.com]
...
> Hi Derek,
> 
> On 06/12/2018 06:56 PM, White, Derek wrote:
> > Hi Swati, Gustavo,
> >
> > I’m not the best qualified to review the change – I just reported the issue
> as a JDK bug!
> >
> > I’d be happy to test a fix but I’m having trouble following the patch. Did
> Gustavo post a patch to your patch, or is that a full independent patch?
> 
> Yes, the idea was that you could help on testing it against JDK-8189922.
> Swati's initial report on this thread was accompanied with a simple way to
> test the issue he reported. You said it was related to bug JDK-8189922 but I
> can't see a simple way to test it as you reported. Besides that I assumed that
> you tested it on arm64, so I can't test it myself (I don't have such a
> hardware). Btw, if you could provide some numactl -H information I would
> be glad.


OK, here's a test case:
$ numactl -N 0 -m 0 java -Xlog:gc*=info -XX:+UseParallelGC -XX:+UseNUMA -version

Before patch, failed output shows 1/2 of Eden being wasted for threads from node that will never allocate:
...
[0.230s][info][gc,heap,exit ]   eden space 524800K, 4% used [0x0000000580100000,0x0000000581580260,0x00000005a0180000)
[0.230s][info][gc,heap,exit ]     lgrp 0 space 262400K, 8% used [0x0000000580100000,0x0000000581580260,0x0000000590140000)
[0.230s][info][gc,heap,exit ]     lgrp 1 space 262400K, 0% used [0x0000000590140000,0x0000000590140000,0x00000005a0180000)
...

After patch, passed output:
...
[0.231s][info][gc,heap,exit ]   eden space 524800K, 8% used [0x0000000580100000,0x0000000582a00260,0x00000005a0180000)
... (no lgrps)

Open questions - still a bug?
1) What should JVM do if cpu node is bound, but not memory is bound? Even with patch, JVM wastes memory because it sets aside part of Eden for threads that can never run on other node.
  - numactl -N 0 java -Xlog:gc*=info -XX:+UseParallelGC -XX:+UseNUMA -version
  - My expectation was that it would act as if membind is set. But I'm not an expert.
  - What do containers do under the hood? Would they ever bind cpus and NOT memory?
2) What should JVM do if cpu node is bound, and numactl --localalloc specified? Even with patch, JVM wastes memory.
  - numactl -N 0 --localalloc java -Xlog:gc*=info -XX:+UseParallelGC -XX:+UseNUMA -version
  - My expectation was that "--localalloc" would be identical to setting membind for all of the cpu bound nodes, but I guess it's not. 



FYI - numactl -H:
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 0 size: 128924 MB
node 0 free: 8499 MB
node 1 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
node 1 size: 129011 MB
node 1 free: 7964 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10

> I consider the patch I pointed out as the fourth version of Swati's original
> proposal, it evolved from the reviews so far:
> http://cr.openjdk.java.net/~gromero/8189922/draft/usenuma_v4.patch
> 
> 
> > Also, if you or Gustavo have permissions to post a webrev to
> http://cr.openjdk.java.net/ that would make reviewing a little easier. I’d be
> happy to post a webrev for you if not.
> 
> I was planing to host the webrev after your comments, but feel free to host
> it.

No, you have it covered well, I'll stay out of it.

 - Derek