Huge pages

Wed Apr 10 10:26:15 PDT 2013

On Tue, Apr 9, 2013 at 11:03 PM, Tiago Stürmer Daitx <
tdaitx at linux.vnet.ibm.com> wrote:
> On Tue, 2013-04-09 at 19:21 +0200, Volker Simonis wrote:
>> I can only imagine that your problem may be perhaps related to the use
>> of compressed pointers (-XX:+UseCompressedOops) which is on by default
>> for our port but off by default for the Zero-VM. As far as I can see,
>> this can lead to a situation where different base addresses are tried
>> for the allocation of the heap and where you may get such a warning
>> although the actual allocation will later succeed for another heap
>> address. But that's only a vague guess. Nevertheless you could try out
>> if switching compressed pointers off (with -XX:-UseCompressedOops)
>> will change something.
>
> You nailed it. Disabling CompressedOops now enables the JVM to make use
> of hugepages.
>
> BTW the Zero VM does work with huge pages when CompressedOops is on (or
> off, it doesn't matter).
>

As Goetz wrote already, Zero doesn't support CompressedOops. It silently
switches them off, even if you use -XX:+UseCompressedOops. You can verify
this with -XX:+PrintFlagsFinal.

>> Can you please also run your java commands (both the port and the
>> zero-vm) under 'strace -f -e trace=ipc' like this:
>>
>> strace -f -e trace=ipc java  -XX:+UseConcMarkSweepGC -XX:
>> +UseLargePages -XX:+UseSHM -Xmx756m -version
>>
>> and post the IPC-relevant output for further investigation.
>
>
> Zero VM (note: CompressedOops does not matter, the output is the same
> but for the shmid)
>
> [pid 40776] shmget(IPC_PRIVATE, 872415232, IPC_CREAT|SHM_HUGETLB|0600) =
> 458752
> [pid 40776] shmat(458752, 0, 0)         = 0x3effcc000000
> [pid 40776] shmctl(458752, IPC_RMID, 0) = 0
>
>
> PPC (CompressedOops disabled)
>
> [pid 40970] shmget(IPC_PRIVATE, 872415232, IPC_CREAT|SHM_HUGETLB|0600) =
> 524288
> [pid 40970] shmat(524288, 0, 0)         = 0x3effcc000000
> [pid 40970] shmctl(524288, IPC_RMID, 0) = 0
>
>
> PPC (CompressedOops enabled)
>
> [pid 41025] shmget(IPC_PRIVATE, 872415232, IPC_CREAT|SHM_HUGETLB|0600) =
> 557056
> [pid 41025] shmat(557056, 0xcc000000, 0) = 0xcc000000
> [pid 41025] shmctl(557056, IPC_RMID, 0) = 0
>

But this looks good. Is this really the case where you get the "(errno =
16)" warning?

>
> I did some additional testing with hugetlbfs and I'm able to use
> hugepages on both PPC and Zero VM (with CompressedOops either on or off)
> by running:
>
> HUGETLB_PATH=/var/lib/hugetlbfs/global/pagesize-16MB hugectl
> --force-preload --shm java -XX:+UseConcMarkSweepGC -XX:+UseLargePages
> -XX:+UseSHM -Xmx756
> m -showversion -jar SPECjvm2008.jar -wt 120 -it 300 --parseJvmArgs -i 1
> --peak
>

I don't really get this. Can you please explain the commands before the
java call in more detail.

>
> Neither Zero VM nor PPC works with +UseHugeTLBFS though (even when
> running hugeadm without --shm).
>

Please notice that -XX:+UseHugeTLBFS and -XX:+UseSHM are mutually
exclusive. See os::large_page_init() in os_linux.cpp:

  UseHugeTLBFS = UseHugeTLBFS &&
                 Linux::hugetlbfs_sanity_check(warn_on_failure,
_large_page_size);

  if (UseHugeTLBFS)
    UseSHM = false;

  UseLargePages = UseHugeTLBFS || UseSHM;

The logic is to try UseHugeTLBFS first. If that works, UseSHM will be
switched off.

That said, I think I found out why UseHugeTLBFS doesn't work on Linux/PPC:

The HotSpot logic of allocating the heap is as follows:
- if 'UseSHM' is active it creates and attaches one single SHM-region for
the heap. It may be that you don't have enough continuous huge page memory
available if you get errors for great heaps with  'UseSHM'. I don't know a
real solution for this but just read in various blogs that it may be
usefull to allocate the huge memory with at boot time.
- if 'UseHugeTLBFS' is active it first mmaps the complete heap with
'MAP_NORESERVE' and later commits parts of the heap as needed with
'MAP_FIXED' but without 'MAP_NORESERVE'.

Now here's the difference between x86 and ppc. For x86 it makes no
difference that the mmap with 'MAP_NORESERVE'  does not have the
MAP_HUGETLB flag. It is still possible to commit the memory later on with
MAP_HUGETLB. On the other hand, on ppc mmap refuses to commit memory with
MAP_HUGETLB if it was first mapped without MAP_HUGETLB. I don't know why
this is the case but that's how it works:(

I've attached a small patch for you, which should enable UseHugeTLBFS on
our Linux/PPC port (i.e. you should be able to use -XX:+UseHugeTLBFS
-XX:+UseLargePages) with and without compressed pointers.

Please understand that I'm a little reluctant to check it in right away,
because as it currently works, it affects all Linux platforms. Moreover the
whole heap memory allocation process is quite weird and interacts with a
lot of other features like compressed oops, numa support and even with
implicit null checks.

But I'll be quite happy if you could try it out and tell me your
experience. I would be especially interested if you can observer some
performance improvements for specjvm for example.

By the way, just for reference, large page support was implemented by
Andrew Haley from RedHat:

PING: Linux: Support transparent
hugepages<http://mail.openjdk.java.net/pipermail/hotspot-dev/2011-April/thread.html#4060>
7034464 : Support transparent large pages on
Linux<http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7034464>

but probably nobody has tested it on Linux/PPC :(

>
> The only additional setting (besides yours) that I had to do was to run
> "hugepages --create-global-mounts" (I used global mounts for simplicity,
> any mount my user had access to would suffice).
>

I don't think you need the mounts for using Java with large pages because
the VM only uses shmat/shmget system calls or mmap with MAP_HUGETLB (see
http://www.mjmwired.net/kernel/Documentation/vm/hugetlbpage.txt)

> Let me know if you need anything else. =)
>
> Regards,
> Tiago
>
> --
> Tiago Stürmer Daitx
> tdaitx at linux.vnet.ibm.com
> IBM - Linux Technology Center
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/attachments/20130410/31fa4677/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: UseHugeTLBFS_ppc.patch
Type: application/octet-stream
Size: 3051 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/ppc-aix-port-dev/attachments/20130410/31fa4677/UseHugeTLBFS_ppc-0001.patch 