Huge pages on PPC64

Mon Oct 31 12:28:55 UTC 2016

Hi Volker,

On 28-10-2016 05:39, Volker Simonis wrote:
> Hi Gustavo,
> 
> please find my comments inline:
> 
> On Thu, Oct 27, 2016 at 11:12 PM, Gustavo Romero
> <gromero at linux.vnet.ibm.com> wrote:
>> Hi Volker,
>>
>> On 25-10-2016 05:13, Volker Simonis wrote:
>>> Just a quick question: do you also get the same warnings if you are
>>> running without compressed oops (i.e. -XX:-UseCompressedClassPointers
>>> -XX:-UseCompressedOops)? If not maybe this is related to the fact that
>>> for compressed oops we try to allocate memory in low memory regions.
>>
>> $ java -XX:+PrintCommandLineFlags -Xms256m -Xmx256m -XX:+UseLargePages -XX:-UseCompressedClassPointers -XX:-UseCompressedOops StrictMath_cos
>> -XX:InitialHeapSize=268435456 -XX:MaxHeapSize=268435456 -XX:+PrintCommandLineFlags -XX:-UseCompressedClassPointers -XX:-UseCompressedOops -XX:+UseLargePages -XX:+UseParallelGC
>> OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
>> 0.1709843554185943
>>
>> It fails when trying to allocate 256 MiB starting at 0x0, so definitely a low
>> memory region. If CompressedClassPointers and CompressedOops are disabled, what
>> could be trying to allocate that region?
>>
> 
> Could you please run the VM in gdb to analyze where the allocation
> fails and why?
> You can also check that way that you really get large pages with -XX:+UseSHM.
> 
> Unfortunately I can't help you much here because I don't have a
> machine with your 'huge pages' / 'transparent hug pages' setting. But
> I know that setting up huge pages on Linux such that mmap() +
> MAP_HUGETLB works is really tricky.

Apparently the allocation at no specific address (i.e. 0x0, thanks Thomas!) is
related to the GC:

(gdb) info b
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   <PENDING>  warn_on_large_pages_failure
(gdb) c
Continuing.
[New Thread 0x3fffb6d9f1a0 (LWP 3626)]
-XX:InitialHeapSize=268435456 -XX:MaxHeapSize=268435456 -XX:+PrintCommandLineFlags -XX:-UseCompressedClassPointers -XX:-UseCompressedOops -XX:+UseLargePages -XX:+UseParallelGC
[Switching to Thread 0x3fffb6d9f1a0 (LWP 3626)]

Breakpoint 1, os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
3505    ./src/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory.
(gdb) bt
#0  os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
#1  0x00003fffb796e350 in os::Linux::reserve_memory_special_huge_tlbfs (exec=<optimized out>, req_addr=0x0, alignment=<optimized out>, bytes=268435456) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3541
#2  os::reserve_memory_special (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3553
#3  0x00003fffb7b7651c in ReservedSpace::initialize (executable=false, noaccess_prefix=0, requested_address=0x0, large=true, alignment=<optimized out>, size=268435456, this=0x3fffb6d9def8)
    at ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:157
#4  ReservedSpace::ReservedSpace (noaccess_prefix=0, requested_address=0x0, large=true, alignment=33554432, size=<optimized out>, this=0x3fffb6d9def8) at
./src/hotspot/src/share/vm/runtime/virtualspace.cpp:79
#5  ReservedHeapSpace::ReservedHeapSpace (this=0x3fffb6d9def8, size=<optimized out>, alignment=33554432, large=<optimized out>, requested_address=0x0) at
./src/hotspot/src/share/vm/runtime/virtualspace.cpp:344
#6  0x00003fffb7b2ee00 in Universe::reserve_heap (heap_size=<optimized out>, alignment=33554432) at ./src/hotspot/src/share/vm/memory/universe.cpp:942
#7  0x00003fffb799caf8 in ParallelScavengeHeap::initialize (this=0x3fffb0016e70) at ./src/hotspot/src/share/vm/gc_implementation/parallelScavenge/parallelScavengeHeap.cpp:64
#8  0x00003fffb7b2f8c0 in Universe::initialize_heap () at ./src/hotspot/src/share/vm/memory/universe.cpp:843
#9  0x00003fffb7b2fcf8 in universe_init () at ./src/hotspot/src/share/vm/memory/universe.cpp:652
#10 0x00003fffb76222d4 in init_globals () at ./src/hotspot/src/share/vm/runtime/init.cpp:104
#11 0x00003fffb7b0849c in Threads::create_vm (args=<optimized out>, canTryAgain=0x3fffb6d9e5af) at ./src/hotspot/src/share/vm/runtime/thread.cpp:3421
#12 0x00003fffb76ce19c in JNI_CreateJavaVM (vm=0x3fffb6d9e690, penv=0x3fffb6d9e698, args=<optimized out>) at ./src/hotspot/src/share/vm/prims/jni.cpp:5220
#13 0x00003fffb7ed31dc in InitializeJVM (ifn=<synthetic pointer>, penv=<optimized out>, pvm=<optimized out>) at ./src/jdk/src/share/bin/java.c:1215
#14 JavaMain (_args=<optimized out>) at ./src/jdk/src/share/bin/java.c:376
#15 0x00003fffb7ed87a8 in call_continuation (_args=<optimized out>) at ./src/jdk/src/solaris/bin/java_md_solinux.c:1034
#16 0x00003fffb7f4809c in start_thread (arg=0x3fffb6d9f1a0) at pthread_create.c:335
#17 0x00003fffb7df1344 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:94
(gdb) n
OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
3513    in ./src/hotspot/src/os/linux/vm/os_linux.cpp
(gdb) !java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2-b14)
OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)

Hence, if we use +UseSerialGC:

(gdb) c
Continuing.
[New Thread 0x3fffb6d9f1a0 (LWP 3842)]
-XX:InitialHeapSize=268435456 -XX:MaxHeapSize=268435456 -XX:+PrintCommandLineFlags -XX:-UseCompressedClassPointers -XX:-UseCompressedOops -XX:+UseLargePages -XX:+UseSerialGC
[Switching to Thread 0x3fffb6d9f1a0 (LWP 3842)]

Breakpoint 1, os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
3505    ./src/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory.
(gdb) bt
#0  os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
#1  0x00003fffb796e350 in os::Linux::reserve_memory_special_huge_tlbfs (exec=<optimized out>, req_addr=0x0, alignment=<optimized out>, bytes=268435456) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3541
#2  os::reserve_memory_special (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3553
#3  0x00003fffb7b7651c in ReservedSpace::initialize (executable=false, noaccess_prefix=0, requested_address=0x0, large=true, alignment=<optimized out>, size=268435456, this=0x3fffb6d9ddc8)
    at ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:157
#4  ReservedSpace::ReservedSpace (noaccess_prefix=0, requested_address=0x0, large=true, alignment=33554432, size=<optimized out>, this=0x3fffb6d9ddc8) at
./src/hotspot/src/share/vm/runtime/virtualspace.cpp:79
#5  ReservedHeapSpace::ReservedHeapSpace (this=0x3fffb6d9ddc8, size=<optimized out>, alignment=33554432, large=<optimized out>, requested_address=0x0) at
./src/hotspot/src/share/vm/runtime/virtualspace.cpp:344
#6  0x00003fffb7b2ee00 in Universe::reserve_heap (heap_size=<optimized out>, alignment=33554432) at ./src/hotspot/src/share/vm/memory/universe.cpp:942
#7  0x00003fffb75a83f0 in GenCollectedHeap::allocate (this=0x3fffb0016fb0, alignment=33554432, _total_reserved=0x3fffb6d9e0c8, _n_covered_regions=0x3fffb6d9e0c4, heap_rs=0x3fffb6d9e0d0)
    at ./src/hotspot/src/share/vm/memory/genCollectedHeap.cpp:200
#8  0x00003fffb75aae58 in GenCollectedHeap::initialize (this=0x3fffb0016fb0) at ./src/hotspot/src/share/vm/memory/genCollectedHeap.cpp:124
#9  0x00003fffb7b2f8c0 in Universe::initialize_heap () at ./src/hotspot/src/share/vm/memory/universe.cpp:843
#10 0x00003fffb7b2fcf8 in universe_init () at ./src/hotspot/src/share/vm/memory/universe.cpp:652
#11 0x00003fffb76222d4 in init_globals () at ./src/hotspot/src/share/vm/runtime/init.cpp:104
#12 0x00003fffb7b0849c in Threads::create_vm (args=<optimized out>, canTryAgain=0x3fffb6d9e5af) at ./src/hotspot/src/share/vm/runtime/thread.cpp:3421
#13 0x00003fffb76ce19c in JNI_CreateJavaVM (vm=0x3fffb6d9e690, penv=0x3fffb6d9e698, args=<optimized out>) at ./src/hotspot/src/share/vm/prims/jni.cpp:5220
#14 0x00003fffb7ed31dc in InitializeJVM (ifn=<synthetic pointer>, penv=<optimized out>, pvm=<optimized out>) at ./src/jdk/src/share/bin/java.c:1215
#15 JavaMain (_args=<optimized out>) at ./src/jdk/src/share/bin/java.c:376
#16 0x00003fffb7ed87a8 in call_continuation (_args=<optimized out>) at ./src/jdk/src/solaris/bin/java_md_solinux.c:1034
#17 0x00003fffb7f4809c in start_thread (arg=0x3fffb6d9f1a0) at pthread_create.c:335
#18 0x00003fffb7df1344 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:94
(gdb) n
OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
3513    in ./src/hotspot/src/os/linux/vm/os_linux.cpp

and if we use +UseG1GC:

Breakpoint 1, os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
3505    ./src/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory.
(gdb) bt
#0  os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
#1  0x00003fffb796e350 in os::Linux::reserve_memory_special_huge_tlbfs (exec=<optimized out>, req_addr=0x0, alignment=<optimized out>, bytes=268435456) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3541
#2  os::reserve_memory_special (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3553
#3  0x00003fffb7b7651c in ReservedSpace::initialize (executable=false, noaccess_prefix=0, requested_address=0x0, large=true, alignment=<optimized out>, size=268435456, this=0x3fffb6d9de78)
    at ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:157
#4  ReservedSpace::ReservedSpace (noaccess_prefix=0, requested_address=0x0, large=true, alignment=33554432, size=<optimized out>, this=0x3fffb6d9de78) at
./src/hotspot/src/share/vm/runtime/virtualspace.cpp:79
#5  ReservedHeapSpace::ReservedHeapSpace (this=0x3fffb6d9de78, size=<optimized out>, alignment=33554432, large=<optimized out>, requested_address=0x0) at
./src/hotspot/src/share/vm/runtime/virtualspace.cpp:344
#6  0x00003fffb7b2ee00 in Universe::reserve_heap (heap_size=<optimized out>, alignment=33554432) at ./src/hotspot/src/share/vm/memory/universe.cpp:942
#7  0x00003fffb7557d48 in G1CollectedHeap::initialize (this=0x3fffb001bb70) at ./src/hotspot/src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp:1975
#8  0x00003fffb7b2f8c0 in Universe::initialize_heap () at ./src/hotspot/src/share/vm/memory/universe.cpp:843
#9  0x00003fffb7b2fcf8 in universe_init () at ./src/hotspot/src/share/vm/memory/universe.cpp:652
#10 0x00003fffb76222d4 in init_globals () at ./src/hotspot/src/share/vm/runtime/init.cpp:104
#11 0x00003fffb7b0849c in Threads::create_vm (args=<optimized out>, canTryAgain=0x3fffb6d9e5af) at ./src/hotspot/src/share/vm/runtime/thread.cpp:3421
#12 0x00003fffb76ce19c in JNI_CreateJavaVM (vm=0x3fffb6d9e690, penv=0x3fffb6d9e698, args=<optimized out>) at ./src/hotspot/src/share/vm/prims/jni.cpp:5220
#13 0x00003fffb7ed31dc in InitializeJVM (ifn=<synthetic pointer>, penv=<optimized out>, pvm=<optimized out>) at ./src/jdk/src/share/bin/java.c:1215
#14 JavaMain (_args=<optimized out>) at ./src/jdk/src/share/bin/java.c:376
#15 0x00003fffb7ed87a8 in call_continuation (_args=<optimized out>) at ./src/jdk/src/solaris/bin/java_md_solinux.c:1034
#16 0x00003fffb7f4809c in start_thread (arg=0x3fffb6d9f1a0) at pthread_create.c:335
#17 0x00003fffb7df1344 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:94
(gdb) n
OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
3513    in ./src/hotspot/src/os/linux/vm/os_linux.cpp

So there is no way to deal with it I guess...

But I tried SPECjbb2005 with THP as Goetz suggested and all went OK. I tried
also with shared memory allocation (+UseLargePages +UseSHM) and all went OK
as well.

So back to my initial question (given that +UseLargePages is indeed comprised of
three alternative methods to allocate space with HP - on Linux), I would say
that Huge Pages on PPC64 JVM is in good shape regarding +UseSHM and
+UseTransparentHugePages, but +UseHugeTLBFS has some issues not worth to care
since THP and SHM are OK. Is it sensible?

Thank you!

Regards,
Gustavo