Huge pages on PPC64
Thomas Stüfe
thomas.stuefe at gmail.com
Wed Nov 2 10:27:47 UTC 2016
Hi Volker, Gustavo,
On Wed, Nov 2, 2016 at 11:01 AM, Volker Simonis <volker.simonis at gmail.com>
wrote:
> Hi Gustavo,
>
> one thing you could still check is why mmap fails with errno=16.
> According to /usr/include/asm-generic/errno-base.h this is EBUSY but
> unfortunately the man page for mmap doesn't document EBUSY as a valid
> error code for mmap. As you seem to work for a team involved in Linux
> development (at least according to you e-mail adress :) maybe you can
> find out what EBUSY means for mmap with MAP_HUGETLB.
>
> But wait, I've just found this old mail thread [1] with Tiago (one of
> your former colleagues):
>
> That time [2] I found out that:
> ...
> if 'UseHugeTLBFS' is active it first mmaps the complete heap with
> 'MAP_NORESERVE' and later commits parts of the heap as needed with
> 'MAP_FIXED' but without 'MAP_NORESERVE'.
>
> Now here's the difference between x86 and ppc. For x86 it makes no
> difference that the mmap with 'MAP_NORESERVE' does not have the
> MAP_HUGETLB flag. It is still possible to commit the memory later on with
> MAP_HUGETLB. On the other hand, on ppc mmap refuses to commit memory with
> MAP_HUGETLB if it was first mapped without MAP_HUGETLB. I don't know why
> this is the case but that's how it works:(
> ...
>
> In a later mail, Tiago explained why this is so [3]:
> ...
> 'On ppc64, you can only have one page size in a "slice". Slices are
> 256M below 4G, then 4G-1TB then 1TB slices above that. Since the
> MAP_NORESERVE is not a hugepage mapping, it is not being placed to
> accomodate these restrictions. Overmapping with MAP_FIXED is then
> failing.'
> ...
>
> Re-reading that old mail thread, I'm pretty sure now that this is the
> actual reason why you see the problems with UseHugeTLBFS on
> Linux/ppc64.
>
> As I wrote in that thread, this could be fixed [4], but it is probably
> not trivial:
> ...
> I think under these circumstances it will be not so easy to make huge
> pages work with the current way how the HotSpot VM allocates memory
> (i.e mmapping huge memory areas with MAP_NORESERVE and later
> committing them partially and independently of each other with
> MAP_FIXED|MAP_HUGETLB).
>
> This would probably require a bigger rework of the allocation code. I
> think I have some basic ideas how this can be done - it requires a
> bookkeeping for every region to remember if it should use huge pages
> or not. Unfortunaltey it's notoriously hard to bring such shared code
> changes into the HotSpot main code line, but perhaps this will become
> easier once our port is part of the HotSpot main code line.
> ...
>
>
Volker, excellent find! :)
Note that we have a workaround for a similar problem in AIX: the problem
that you need to keep meta information associated with a block allocated
with os::reserve_memory() for later use with the other memory functions. On
AIX, it is the question if the memory range has been allocated via shmget
or mmap.
The code in the AIX port is in os_aix.cpp, see the "vmembk_..." functions.
Basically, we keep meta information in a linked list inside the porting
layer.
But this is clunky and ugly. The underlying problem is that the
os::reserve_memory() API cannot be easily implemented by porters for cases
where you need to keep additional meta information for a chunk of memory.
To solve this cleanly, this would have to be solved at API level.
The typical way to do this in C is to hand back to the caller not a raw
pointer but a pointer to an opaque structure which the porter then can use
to store information in. Caller then hands this structure back in into any
subsequent memory functions. Basically, a Handle.
For instance:
struct memory_handle_t { char* pointer; .... (opaque porting specific data
like mmap flags used for creation) ... };
memory_handle_t * os_reserve_memory(size); (allocate memory_handle_t
structure and fill in pointer and anything else)
bool os::commit_memory(memory_handle_t* memory);
bool os::release_memory(memory_handle_t* memory); ( free memory_handle_t)
Users of os::reserve_memory would have to use memory_handle_t.pointer
member to get the raw data pointer. Because that would affect all users of
os::reserve_memory, it would make ripples all over shared code, but I think
this would be a pragmatic and clean solution. I think this would be a nice
enhancement for 10. If you think it makes sense, I can open an issue in jbs.
> You're welcome to try your luck :)
>
> Regards,
> Volker
>
> [1] http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/
> 2013-April/thread.html#385
> [2] http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/
> 2013-April/000397.html
> [3] http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/
> 2013-April/000445.html
> [4] http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/
> 2013-April/000447.html
>
>
Kind Regards, Thomas
> On Mon, Oct 31, 2016 at 1:28 PM, Gustavo Romero
> <gromero at linux.vnet.ibm.com> wrote:
> > Hi Volker,
> >
> > On 28-10-2016 05:39, Volker Simonis wrote:
> >> Hi Gustavo,
> >>
> >> please find my comments inline:
> >>
> >> On Thu, Oct 27, 2016 at 11:12 PM, Gustavo Romero
> >> <gromero at linux.vnet.ibm.com> wrote:
> >>> Hi Volker,
> >>>
> >>> On 25-10-2016 05:13, Volker Simonis wrote:
> >>>> Just a quick question: do you also get the same warnings if you are
> >>>> running without compressed oops (i.e. -XX:-UseCompressedClassPointers
> >>>> -XX:-UseCompressedOops)? If not maybe this is related to the fact that
> >>>> for compressed oops we try to allocate memory in low memory regions.
> >>>
> >>> $ java -XX:+PrintCommandLineFlags -Xms256m -Xmx256m -XX:+UseLargePages
> -XX:-UseCompressedClassPointers -XX:-UseCompressedOops StrictMath_cos
> >>> -XX:InitialHeapSize=268435456 -XX:MaxHeapSize=268435456
> -XX:+PrintCommandLineFlags -XX:-UseCompressedClassPointers
> -XX:-UseCompressedOops -XX:+UseLargePages -XX:+UseParallelGC
> >>> OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory
> req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
> >>> 0.1709843554185943
> >>>
> >>> It fails when trying to allocate 256 MiB starting at 0x0, so
> definitely a low
> >>> memory region. If CompressedClassPointers and CompressedOops are
> disabled, what
> >>> could be trying to allocate that region?
> >>>
> >>
> >> Could you please run the VM in gdb to analyze where the allocation
> >> fails and why?
> >> You can also check that way that you really get large pages with
> -XX:+UseSHM.
> >>
> >> Unfortunately I can't help you much here because I don't have a
> >> machine with your 'huge pages' / 'transparent hug pages' setting. But
> >> I know that setting up huge pages on Linux such that mmap() +
> >> MAP_HUGETLB works is really tricky.
> >
> > Apparently the allocation at no specific address (i.e. 0x0, thanks
> Thomas!) is
> > related to the GC:
> >
> > (gdb) info b
> > Num Type Disp Enb Address What
> > 1 breakpoint keep y <PENDING> warn_on_large_pages_failure
> > (gdb) c
> > Continuing.
> > [New Thread 0x3fffb6d9f1a0 (LWP 3626)]
> > -XX:InitialHeapSize=268435456 -XX:MaxHeapSize=268435456
> -XX:+PrintCommandLineFlags -XX:-UseCompressedClassPointers
> -XX:-UseCompressedOops -XX:+UseLargePages -XX:+UseParallelGC
> > [Switching to Thread 0x3fffb6d9f1a0 (LWP 3626)]
> >
> > Breakpoint 1, os::Linux::reserve_memory_special_huge_tlbfs_mixed
> (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized
> out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> > 3505 ./src/hotspot/src/os/linux/vm/os_linux.cpp: No such file or
> directory.
> > (gdb) bt
> > #0 os::Linux::reserve_memory_special_huge_tlbfs_mixed
> (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized
> out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> > #1 0x00003fffb796e350 in os::Linux::reserve_memory_special_huge_tlbfs
> (exec=<optimized out>, req_addr=0x0, alignment=<optimized out>,
> bytes=268435456) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3541
> > #2 os::reserve_memory_special (bytes=268435456, alignment=<optimized
> out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/
> os_linux.cpp:3553
> > #3 0x00003fffb7b7651c in ReservedSpace::initialize (executable=false,
> noaccess_prefix=0, requested_address=0x0, large=true, alignment=<optimized
> out>, size=268435456, this=0x3fffb6d9def8)
> > at ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:157
> > #4 ReservedSpace::ReservedSpace (noaccess_prefix=0,
> requested_address=0x0, large=true, alignment=33554432, size=<optimized
> out>, this=0x3fffb6d9def8) at
> > ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:79
> > #5 ReservedHeapSpace::ReservedHeapSpace (this=0x3fffb6d9def8,
> size=<optimized out>, alignment=33554432, large=<optimized out>,
> requested_address=0x0) at
> > ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:344
> > #6 0x00003fffb7b2ee00 in Universe::reserve_heap (heap_size=<optimized
> out>, alignment=33554432) at ./src/hotspot/src/share/vm/
> memory/universe.cpp:942
> > #7 0x00003fffb799caf8 in ParallelScavengeHeap::initialize
> (this=0x3fffb0016e70) at ./src/hotspot/src/share/vm/gc_implementation/
> parallelScavenge/parallelScavengeHeap.cpp:64
> > #8 0x00003fffb7b2f8c0 in Universe::initialize_heap () at
> ./src/hotspot/src/share/vm/memory/universe.cpp:843
> > #9 0x00003fffb7b2fcf8 in universe_init () at ./src/hotspot/src/share/vm/
> memory/universe.cpp:652
> > #10 0x00003fffb76222d4 in init_globals () at ./src/hotspot/src/share/vm/
> runtime/init.cpp:104
> > #11 0x00003fffb7b0849c in Threads::create_vm (args=<optimized out>,
> canTryAgain=0x3fffb6d9e5af) at ./src/hotspot/src/share/vm/
> runtime/thread.cpp:3421
> > #12 0x00003fffb76ce19c in JNI_CreateJavaVM (vm=0x3fffb6d9e690,
> penv=0x3fffb6d9e698, args=<optimized out>) at ./src/hotspot/src/share/vm/
> prims/jni.cpp:5220
> > #13 0x00003fffb7ed31dc in InitializeJVM (ifn=<synthetic pointer>,
> penv=<optimized out>, pvm=<optimized out>) at ./src/jdk/src/share/bin/java.
> c:1215
> > #14 JavaMain (_args=<optimized out>) at ./src/jdk/src/share/bin/java.
> c:376
> > #15 0x00003fffb7ed87a8 in call_continuation (_args=<optimized out>) at
> ./src/jdk/src/solaris/bin/java_md_solinux.c:1034
> > #16 0x00003fffb7f4809c in start_thread (arg=0x3fffb6d9f1a0) at
> pthread_create.c:335
> > #17 0x00003fffb7df1344 in clone () at ../sysdeps/unix/sysv/linux/
> powerpc/powerpc64/clone.S:94
> > (gdb) n
> > OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory
> req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
> > 3513 in ./src/hotspot/src/os/linux/vm/os_linux.cpp
> > (gdb) !java -version
> > openjdk version "1.8.0_111"
> > OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2-b14)
> > OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
> >
> > Hence, if we use +UseSerialGC:
> >
> > (gdb) c
> > Continuing.
> > [New Thread 0x3fffb6d9f1a0 (LWP 3842)]
> > -XX:InitialHeapSize=268435456 -XX:MaxHeapSize=268435456
> -XX:+PrintCommandLineFlags -XX:-UseCompressedClassPointers
> -XX:-UseCompressedOops -XX:+UseLargePages -XX:+UseSerialGC
> > [Switching to Thread 0x3fffb6d9f1a0 (LWP 3842)]
> >
> > Breakpoint 1, os::Linux::reserve_memory_special_huge_tlbfs_mixed
> (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized
> out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> > 3505 ./src/hotspot/src/os/linux/vm/os_linux.cpp: No such file or
> directory.
> > (gdb) bt
> > #0 os::Linux::reserve_memory_special_huge_tlbfs_mixed
> (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized
> out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> > #1 0x00003fffb796e350 in os::Linux::reserve_memory_special_huge_tlbfs
> (exec=<optimized out>, req_addr=0x0, alignment=<optimized out>,
> bytes=268435456) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3541
> > #2 os::reserve_memory_special (bytes=268435456, alignment=<optimized
> out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/
> os_linux.cpp:3553
> > #3 0x00003fffb7b7651c in ReservedSpace::initialize (executable=false,
> noaccess_prefix=0, requested_address=0x0, large=true, alignment=<optimized
> out>, size=268435456, this=0x3fffb6d9ddc8)
> > at ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:157
> > #4 ReservedSpace::ReservedSpace (noaccess_prefix=0,
> requested_address=0x0, large=true, alignment=33554432, size=<optimized
> out>, this=0x3fffb6d9ddc8) at
> > ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:79
> > #5 ReservedHeapSpace::ReservedHeapSpace (this=0x3fffb6d9ddc8,
> size=<optimized out>, alignment=33554432, large=<optimized out>,
> requested_address=0x0) at
> > ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:344
> > #6 0x00003fffb7b2ee00 in Universe::reserve_heap (heap_size=<optimized
> out>, alignment=33554432) at ./src/hotspot/src/share/vm/
> memory/universe.cpp:942
> > #7 0x00003fffb75a83f0 in GenCollectedHeap::allocate
> (this=0x3fffb0016fb0, alignment=33554432, _total_reserved=0x3fffb6d9e0c8,
> _n_covered_regions=0x3fffb6d9e0c4, heap_rs=0x3fffb6d9e0d0)
> > at ./src/hotspot/src/share/vm/memory/genCollectedHeap.cpp:200
> > #8 0x00003fffb75aae58 in GenCollectedHeap::initialize
> (this=0x3fffb0016fb0) at ./src/hotspot/src/share/vm/
> memory/genCollectedHeap.cpp:124
> > #9 0x00003fffb7b2f8c0 in Universe::initialize_heap () at
> ./src/hotspot/src/share/vm/memory/universe.cpp:843
> > #10 0x00003fffb7b2fcf8 in universe_init () at ./src/hotspot/src/share/vm/
> memory/universe.cpp:652
> > #11 0x00003fffb76222d4 in init_globals () at ./src/hotspot/src/share/vm/
> runtime/init.cpp:104
> > #12 0x00003fffb7b0849c in Threads::create_vm (args=<optimized out>,
> canTryAgain=0x3fffb6d9e5af) at ./src/hotspot/src/share/vm/
> runtime/thread.cpp:3421
> > #13 0x00003fffb76ce19c in JNI_CreateJavaVM (vm=0x3fffb6d9e690,
> penv=0x3fffb6d9e698, args=<optimized out>) at ./src/hotspot/src/share/vm/
> prims/jni.cpp:5220
> > #14 0x00003fffb7ed31dc in InitializeJVM (ifn=<synthetic pointer>,
> penv=<optimized out>, pvm=<optimized out>) at ./src/jdk/src/share/bin/java.
> c:1215
> > #15 JavaMain (_args=<optimized out>) at ./src/jdk/src/share/bin/java.
> c:376
> > #16 0x00003fffb7ed87a8 in call_continuation (_args=<optimized out>) at
> ./src/jdk/src/solaris/bin/java_md_solinux.c:1034
> > #17 0x00003fffb7f4809c in start_thread (arg=0x3fffb6d9f1a0) at
> pthread_create.c:335
> > #18 0x00003fffb7df1344 in clone () at ../sysdeps/unix/sysv/linux/
> powerpc/powerpc64/clone.S:94
> > (gdb) n
> > OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory
> req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
> > 3513 in ./src/hotspot/src/os/linux/vm/os_linux.cpp
> >
> > and if we use +UseG1GC:
> >
> > Breakpoint 1, os::Linux::reserve_memory_special_huge_tlbfs_mixed
> (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized
> out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> > 3505 ./src/hotspot/src/os/linux/vm/os_linux.cpp: No such file or
> directory.
> > (gdb) bt
> > #0 os::Linux::reserve_memory_special_huge_tlbfs_mixed
> (bytes=268435456, alignment=<optimized out>, req_addr=0x0, exec=<optimized
> out>) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3505
> > #1 0x00003fffb796e350 in os::Linux::reserve_memory_special_huge_tlbfs
> (exec=<optimized out>, req_addr=0x0, alignment=<optimized out>,
> bytes=268435456) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:3541
> > #2 os::reserve_memory_special (bytes=268435456, alignment=<optimized
> out>, req_addr=0x0, exec=<optimized out>) at ./src/hotspot/src/os/linux/vm/
> os_linux.cpp:3553
> > #3 0x00003fffb7b7651c in ReservedSpace::initialize (executable=false,
> noaccess_prefix=0, requested_address=0x0, large=true, alignment=<optimized
> out>, size=268435456, this=0x3fffb6d9de78)
> > at ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:157
> > #4 ReservedSpace::ReservedSpace (noaccess_prefix=0,
> requested_address=0x0, large=true, alignment=33554432, size=<optimized
> out>, this=0x3fffb6d9de78) at
> > ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:79
> > #5 ReservedHeapSpace::ReservedHeapSpace (this=0x3fffb6d9de78,
> size=<optimized out>, alignment=33554432, large=<optimized out>,
> requested_address=0x0) at
> > ./src/hotspot/src/share/vm/runtime/virtualspace.cpp:344
> > #6 0x00003fffb7b2ee00 in Universe::reserve_heap (heap_size=<optimized
> out>, alignment=33554432) at ./src/hotspot/src/share/vm/
> memory/universe.cpp:942
> > #7 0x00003fffb7557d48 in G1CollectedHeap::initialize
> (this=0x3fffb001bb70) at ./src/hotspot/src/share/vm/gc_implementation/g1/
> g1CollectedHeap.cpp:1975
> > #8 0x00003fffb7b2f8c0 in Universe::initialize_heap () at
> ./src/hotspot/src/share/vm/memory/universe.cpp:843
> > #9 0x00003fffb7b2fcf8 in universe_init () at ./src/hotspot/src/share/vm/
> memory/universe.cpp:652
> > #10 0x00003fffb76222d4 in init_globals () at ./src/hotspot/src/share/vm/
> runtime/init.cpp:104
> > #11 0x00003fffb7b0849c in Threads::create_vm (args=<optimized out>,
> canTryAgain=0x3fffb6d9e5af) at ./src/hotspot/src/share/vm/
> runtime/thread.cpp:3421
> > #12 0x00003fffb76ce19c in JNI_CreateJavaVM (vm=0x3fffb6d9e690,
> penv=0x3fffb6d9e698, args=<optimized out>) at ./src/hotspot/src/share/vm/
> prims/jni.cpp:5220
> > #13 0x00003fffb7ed31dc in InitializeJVM (ifn=<synthetic pointer>,
> penv=<optimized out>, pvm=<optimized out>) at ./src/jdk/src/share/bin/java.
> c:1215
> > #14 JavaMain (_args=<optimized out>) at ./src/jdk/src/share/bin/java.
> c:376
> > #15 0x00003fffb7ed87a8 in call_continuation (_args=<optimized out>) at
> ./src/jdk/src/solaris/bin/java_md_solinux.c:1034
> > #16 0x00003fffb7f4809c in start_thread (arg=0x3fffb6d9f1a0) at
> pthread_create.c:335
> > #17 0x00003fffb7df1344 in clone () at ../sysdeps/unix/sysv/linux/
> powerpc/powerpc64/clone.S:94
> > (gdb) n
> > OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory
> req_addr: 0x0000000000000000 bytes: 268435456 (errno = 16).
> > 3513 in ./src/hotspot/src/os/linux/vm/os_linux.cpp
> >
> > So there is no way to deal with it I guess...
> >
> > But I tried SPECjbb2005 with THP as Goetz suggested and all went OK. I
> tried
> > also with shared memory allocation (+UseLargePages +UseSHM) and all went
> OK
> > as well.
> >
> > So back to my initial question (given that +UseLargePages is indeed
> comprised of
> > three alternative methods to allocate space with HP - on Linux), I would
> say
> > that Huge Pages on PPC64 JVM is in good shape regarding +UseSHM and
> > +UseTransparentHugePages, but +UseHugeTLBFS has some issues not worth to
> care
> > since THP and SHM are OK. Is it sensible?
> >
> > Thank you!
> >
> > Regards,
> > Gustavo
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/attachments/20161102/a98a2447/attachment-0001.html>
More information about the ppc-aix-port-dev
mailing list