Huge pages on PPC64

Volker Simonis volker.simonis at gmail.com
Wed Nov 2 10:01:55 UTC 2016 

Hi Gustavo,

one thing you could still check is why mmap fails with errno=16.
According to /usr/include/asm-generic/errno-base.h this is EBUSY but
unfortunately the man page for mmap doesn't document EBUSY as a valid
error code for mmap. As you seem to work for a team involved in Linux
development (at least according to you e-mail adress :) maybe you can
find out what EBUSY means for mmap with MAP_HUGETLB.

But wait, I've just found this old mail thread [1] with Tiago (one of
your former colleagues):

That time [2] I found out that:
...
if 'UseHugeTLBFS' is active it first mmaps the complete heap with
'MAP_NORESERVE' and later commits parts of the heap as needed with
'MAP_FIXED' but without 'MAP_NORESERVE'.

Now here's the difference between x86 and ppc. For x86 it makes no
difference that the mmap with 'MAP_NORESERVE'  does not have the
MAP_HUGETLB flag. It is still possible to commit the memory later on with
MAP_HUGETLB. On the other hand, on ppc mmap refuses to commit memory with
MAP_HUGETLB if it was first mapped without MAP_HUGETLB. I don't know why
this is the case but that's how it works:(
...

In a later mail, Tiago explained why this is so [3]:
...
'On ppc64, you can only have one page size in a "slice".  Slices are
256M below 4G, then 4G-1TB then 1TB slices above that.  Since the
MAP_NORESERVE is not a hugepage mapping, it is not being placed to
accomodate these restrictions. Overmapping with MAP_FIXED is then
failing.'
...

Re-reading that old mail thread, I'm pretty sure now that this is the
actual reason why you see the problems with UseHugeTLBFS on
Linux/ppc64.

As I wrote in that thread, this could be fixed [4], but it is probably
not trivial:
...
I think under these circumstances it will be not so easy to make huge
pages work with the current way how the HotSpot VM allocates memory
(i.e mmapping huge memory areas with MAP_NORESERVE and later
committing them partially and independently of each other with
MAP_FIXED|MAP_HUGETLB).

This would probably require a bigger rework of the allocation code. I
think I have some basic ideas how this can be done - it requires a
bookkeeping for every region to remember if it should use huge pages
or not. Unfortunaltey it's notoriously hard to bring such shared code
changes into the HotSpot main code line, but perhaps this will become
easier once our port is part of the HotSpot main code line.
...

You're welcome to try your luck :)

Regards,
Volker

[1] http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/2013-April/thread.html#385
[2] http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/2013-April/000397.html
[3] http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/2013-April/000445.html
[4] http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/2013-April/000447.html

On Mon, Oct 31, 2016 at 1:28 PM, Gustavo Romero
<gromero at linux.vnet.ibm.com> wrote:
> Hi Volker,
>
> On 28-10-2016 05:39, Volker Simonis wrote:
>> Hi Gustavo,
>>
>> please find my comments inline:
>>
>> On Thu, Oct 27, 2016 at 11:12 PM, Gustavo Romero
>> <gromero at linux.vnet.ibm.com> wrote:
>>> Hi Volker,
>>>
>>> On 25-10-2016 05:13, Volker Simonis wrote:
>>>> Just a quick question: do you also get the same warnings if you are
>>>> running without compressed oops (i.e. -XX:-UseCompressedClassPointers
>>>> -XX:-UseCompressedOops)? If not maybe this is related to the fact that
>>>> for compressed oops we try to allocate memory in low memory regions.
>>>
>>> $ java -XX:+PrintCommandLineFlags -Xms256m -Xmx256m -XX:+UseLargePages -XX:-UseCompressedClassPointers -XX:-UseCompressedOops StrictMath_cos
>>> -XX:InitialHeapSize=268435456 -XX:MaxHeapSize=268435456 -XX:+PrintCommandLineFlags -XX:-UseCompressedClassPointers -XX:-UseCompressedOops -XX:+UseLargePages -XX:+UseParallelGC
>>> OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
>>> 0.1709843554185943
>>>
>>> It fails when trying to allocate 256 MiB starting at 0x0, so definitely a low
>>> memory region. If CompressedClassPointers and CompressedOops are disabled, what
>>> could be trying to allocate that region?
>>>
>>
>> Could you please run the VM in gdb to analyze where the allocation
>> fails and why?
>> You can also check that way that you really get large pages with -XX:+UseSHM.
>>
>> Unfortunately I can't help you much here because I don't have a
>> machine with your 'huge pages' / 'transparent hug pages' setting. But
>> I know that setting up huge pages on Linux such that mmap() +
>> MAP_HUGETLB works is really tricky.
>
> Apparently the allocation at no specific address (i.e. 0x0, thanks Thomas!) is
> related to the GC:
>
> (gdb) info b
> Num     Type           Disp Enb Address    What
> 1       breakpoint     keep y   <PENDING>  warn_on_large_pages_failure
> (gdb) c
> Continuing.
> [New Thread 0x3fffb6d9f1a0 (LWP 3626)]
> -XX:InitialHeapSize=268435456 -XX:MaxHeapSize=268435456 -XX:+PrintCommandLineFlags -XX:-UseCompressedClassPointers -XX:-UseCompressedOops -XX:+UseLargePages -XX:+UseParallelGC
> [Switching to Thread 0x3fffb6d9f1a0 (LWP 3626)]
>
> Breakpoint 1, os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> 3505    ./src/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory.
> (gdb) bt
> #0  os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> #1  0x00003fffb796e350 in os::Linux::reserve_memory_special_huge_tlbfs (exec=<optimized out>, req_addr=0x0, alignment=<optimized out>, bytes=268435456) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3541
> #2  os::reserve_memory_special (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3553
> #3  0x00003fffb7b7651c in ReservedSpace::initialize (executable=false, noaccess_prefix=0, requested_address=0x0, large=true, alignment=<optimized out>, size=268435456, this=0x3fffb6d9def8)
>     at ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:157
> #4  ReservedSpace::ReservedSpace (noaccess_prefix=0, requested_address=0x0, large=true, alignment=33554432, size=<optimized out>, this=0x3fffb6d9def8) at
> ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:79
> #5  ReservedHeapSpace::ReservedHeapSpace (this=0x3fffb6d9def8, size=<optimized out>, alignment=33554432, large=<optimized out>, requested_address=0x0) at
> ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:344
> #6  0x00003fffb7b2ee00 in Universe::reserve_heap (heap_size=<optimized out>, alignment=33554432) at ./src/hotspot/src/share/vm/memory/universe.cpp:942
> #7  0x00003fffb799caf8 in ParallelScavengeHeap::initialize (this=0x3fffb0016e70) at ./src/hotspot/src/share/vm/gc_implementation/parallelScavenge/parallelScavengeHeap.cpp:64
> #8  0x00003fffb7b2f8c0 in Universe::initialize_heap () at ./src/hotspot/src/share/vm/memory/universe.cpp:843
> #9  0x00003fffb7b2fcf8 in universe_init () at ./src/hotspot/src/share/vm/memory/universe.cpp:652
> #10 0x00003fffb76222d4 in init_globals () at ./src/hotspot/src/share/vm/runtime/init.cpp:104
> #11 0x00003fffb7b0849c in Threads::create_vm (args=<optimized out>, canTryAgain=0x3fffb6d9e5af) at ./src/hotspot/src/share/vm/runtime/thread.cpp:3421
> #12 0x00003fffb76ce19c in JNI_CreateJavaVM (vm=0x3fffb6d9e690, penv=0x3fffb6d9e698, args=<optimized out>) at ./src/hotspot/src/share/vm/prims/jni.cpp:5220
> #13 0x00003fffb7ed31dc in InitializeJVM (ifn=<synthetic pointer>, penv=<optimized out>, pvm=<optimized out>) at ./src/jdk/src/share/bin/java.c:1215
> #14 JavaMain (_args=<optimized out>) at ./src/jdk/src/share/bin/java.c:376
> #15 0x00003fffb7ed87a8 in call_continuation (_args=<optimized out>) at ./src/jdk/src/solaris/bin/java_md_solinux.c:1034
> #16 0x00003fffb7f4809c in start_thread (arg=0x3fffb6d9f1a0) at pthread_create.c:335
> #17 0x00003fffb7df1344 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:94
> (gdb) n
> OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
> 3513    in ./src/hotspot/src/os/linux/vm/os_linux.cpp
> (gdb) !java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>
> Hence, if we use +UseSerialGC:
>
> (gdb) c
> Continuing.
> [New Thread 0x3fffb6d9f1a0 (LWP 3842)]
> -XX:InitialHeapSize=268435456 -XX:MaxHeapSize=268435456 -XX:+PrintCommandLineFlags -XX:-UseCompressedClassPointers -XX:-UseCompressedOops -XX:+UseLargePages -XX:+UseSerialGC
> [Switching to Thread 0x3fffb6d9f1a0 (LWP 3842)]
>
> Breakpoint 1, os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> 3505    ./src/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory.
> (gdb) bt
> #0  os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> #1  0x00003fffb796e350 in os::Linux::reserve_memory_special_huge_tlbfs (exec=<optimized out>, req_addr=0x0, alignment=<optimized out>, bytes=268435456) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3541
> #2  os::reserve_memory_special (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3553
> #3  0x00003fffb7b7651c in ReservedSpace::initialize (executable=false, noaccess_prefix=0, requested_address=0x0, large=true, alignment=<optimized out>, size=268435456, this=0x3fffb6d9ddc8)
>     at ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:157
> #4  ReservedSpace::ReservedSpace (noaccess_prefix=0, requested_address=0x0, large=true, alignment=33554432, size=<optimized out>, this=0x3fffb6d9ddc8) at
> ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:79
> #5  ReservedHeapSpace::ReservedHeapSpace (this=0x3fffb6d9ddc8, size=<optimized out>, alignment=33554432, large=<optimized out>, requested_address=0x0) at
> ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:344
> #6  0x00003fffb7b2ee00 in Universe::reserve_heap (heap_size=<optimized out>, alignment=33554432) at ./src/hotspot/src/share/vm/memory/universe.cpp:942
> #7  0x00003fffb75a83f0 in GenCollectedHeap::allocate (this=0x3fffb0016fb0, alignment=33554432, _total_reserved=0x3fffb6d9e0c8, _n_covered_regions=0x3fffb6d9e0c4, heap_rs=0x3fffb6d9e0d0)
>     at ./src/hotspot/src/share/vm/memory/genCollectedHeap.cpp:200
> #8  0x00003fffb75aae58 in GenCollectedHeap::initialize (this=0x3fffb0016fb0) at ./src/hotspot/src/share/vm/memory/genCollectedHeap.cpp:124
> #9  0x00003fffb7b2f8c0 in Universe::initialize_heap () at ./src/hotspot/src/share/vm/memory/universe.cpp:843
> #10 0x00003fffb7b2fcf8 in universe_init () at ./src/hotspot/src/share/vm/memory/universe.cpp:652
> #11 0x00003fffb76222d4 in init_globals () at ./src/hotspot/src/share/vm/runtime/init.cpp:104
> #12 0x00003fffb7b0849c in Threads::create_vm (args=<optimized out>, canTryAgain=0x3fffb6d9e5af) at ./src/hotspot/src/share/vm/runtime/thread.cpp:3421
> #13 0x00003fffb76ce19c in JNI_CreateJavaVM (vm=0x3fffb6d9e690, penv=0x3fffb6d9e698, args=<optimized out>) at ./src/hotspot/src/share/vm/prims/jni.cpp:5220
> #14 0x00003fffb7ed31dc in InitializeJVM (ifn=<synthetic pointer>, penv=<optimized out>, pvm=<optimized out>) at ./src/jdk/src/share/bin/java.c:1215
> #15 JavaMain (_args=<optimized out>) at ./src/jdk/src/share/bin/java.c:376
> #16 0x00003fffb7ed87a8 in call_continuation (_args=<optimized out>) at ./src/jdk/src/solaris/bin/java_md_solinux.c:1034
> #17 0x00003fffb7f4809c in start_thread (arg=0x3fffb6d9f1a0) at pthread_create.c:335
> #18 0x00003fffb7df1344 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:94
> (gdb) n
> OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
> 3513    in ./src/hotspot/src/os/linux/vm/os_linux.cpp
>
> and if we use +UseG1GC:
>
> Breakpoint 1, os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> 3505    ./src/hotspot/src/os/linux/vm/os_linux.cpp: No such file or directory.
> (gdb) bt
> #0  os::Linux::reserve_memory_special_huge_tlbfs_mixed (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> #1  0x00003fffb796e350 in os::Linux::reserve_memory_special_huge_tlbfs (exec=<optimized out>, req_addr=0x0, alignment=<optimized out>, bytes=268435456) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3541
> #2  os::reserve_memory_special (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3553
> #3  0x00003fffb7b7651c in ReservedSpace::initialize (executable=false, noaccess_prefix=0, requested_address=0x0, large=true, alignment=<optimized out>, size=268435456, this=0x3fffb6d9de78)
>     at ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:157
> #4  ReservedSpace::ReservedSpace (noaccess_prefix=0, requested_address=0x0, large=true, alignment=33554432, size=<optimized out>, this=0x3fffb6d9de78) at
> ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:79
> #5  ReservedHeapSpace::ReservedHeapSpace (this=0x3fffb6d9de78, size=<optimized out>, alignment=33554432, large=<optimized out>, requested_address=0x0) at
> ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:344
> #6  0x00003fffb7b2ee00 in Universe::reserve_heap (heap_size=<optimized out>, alignment=33554432) at ./src/hotspot/src/share/vm/memory/universe.cpp:942
> #7  0x00003fffb7557d48 in G1CollectedHeap::initialize (this=0x3fffb001bb70) at ./src/hotspot/src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp:1975
> #8  0x00003fffb7b2f8c0 in Universe::initialize_heap () at ./src/hotspot/src/share/vm/memory/universe.cpp:843
> #9  0x00003fffb7b2fcf8 in universe_init () at ./src/hotspot/src/share/vm/memory/universe.cpp:652
> #10 0x00003fffb76222d4 in init_globals () at ./src/hotspot/src/share/vm/runtime/init.cpp:104
> #11 0x00003fffb7b0849c in Threads::create_vm (args=<optimized out>, canTryAgain=0x3fffb6d9e5af) at ./src/hotspot/src/share/vm/runtime/thread.cpp:3421
> #12 0x00003fffb76ce19c in JNI_CreateJavaVM (vm=0x3fffb6d9e690, penv=0x3fffb6d9e698, args=<optimized out>) at ./src/hotspot/src/share/vm/prims/jni.cpp:5220
> #13 0x00003fffb7ed31dc in InitializeJVM (ifn=<synthetic pointer>, penv=<optimized out>, pvm=<optimized out>) at ./src/jdk/src/share/bin/java.c:1215
> #14 JavaMain (_args=<optimized out>) at ./src/jdk/src/share/bin/java.c:376
> #15 0x00003fffb7ed87a8 in call_continuation (_args=<optimized out>) at ./src/jdk/src/solaris/bin/java_md_solinux.c:1034
> #16 0x00003fffb7f4809c in start_thread (arg=0x3fffb6d9f1a0) at pthread_create.c:335
> #17 0x00003fffb7df1344 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:94
> (gdb) n
> OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
> 3513    in ./src/hotspot/src/os/linux/vm/os_linux.cpp
>
> So there is no way to deal with it I guess...
>
> But I tried SPECjbb2005 with THP as Goetz suggested and all went OK. I tried
> also with shared memory allocation (+UseLargePages +UseSHM) and all went OK
> as well.
>
> So back to my initial question (given that +UseLargePages is indeed comprised of
> three alternative methods to allocate space with HP - on Linux), I would say
> that Huge Pages on PPC64 JVM is in good shape regarding +UseSHM and
> +UseTransparentHugePages, but +UseHugeTLBFS has some issues not worth to care
> since THP and SHM are OK. Is it sensible?
>
> Thank you!
>
> Regards,
> Gustavo
>

More information about the ppc-aix-port-dev mailing list