Linux large pages: why do we have UseSHM?

[Thomas Stüfe]

Hi David,

thanks a lot, David. This made a lot of things clearer (BTW I wish the
patch links would work for these old bugs in JBS).

So here is what I understand:

UseSHM had been there first.

Then JDK-7034464 introduced Transparent Huge Pages. Those needs
madvise(MADV_HUGEPAGE). For TPH, allocating via mmap(MAP_HUGETLB) was added
alongside the old shmget(). I am not sure why, would TPH not have worked
with shmget()? But anyway, for a while TLBFS was synonymous with TPH.

Later we had https://bugs.openjdk.java.net/browse/JDK-8024838, which
claimed TPH were responsible for performance problems and switched support
for them off by default. This left the mmap(MAP_HUGETLB) allocation intact
but avoided now using TPH. So it gave us the new explicit-TLBFS mode
(UseHugeTLBFS).

Interestingly, one of the reasons JDK-7034464 gave for using mmap() instead
of SHM was that mmap can be committed on demand. That is true (you can call
mmap on every part of a mapping), but later with
https://bugs.openjdk.java.net/browse/JDK-8007074 this capability had been
switched off. Commit on demand would need to mmap(MAP_HUGETLBFS) to commit
and uncommit a mapping, but that may fail if the huge page pool is
exhausted, which would make the mmap call fail, which would introduce a
hole in the reserved space which concurrently running code could
accidentally mmap into... so, in effect, TLBFS is always committed now.

There are still some holes in my understanding. But mainly, I wonder
whether we still need UseSHM. Would love to hear opinions.

Thanks! Thomas

On Sat, Dec 5, 2020 at 6:33 AM David Holmes <david.holmes at oracle.com> wrote:

> Hi Thomas,
>
> On 4/12/2020 4:48 am, Thomas Stüfe wrote:
> > Hi,
> >
> > I wonder why we support two different ways (discounting THPs) to allocate
> > huge pages on Linux.
> >
> > We support allocating huge pages both via System V shm APIs (UseSHM) and
> > via the "newer" mmap Posix APIs (UseHugeTLBFS). The latter is default for
> > +UseLargePages.
>
> We had the UseSHM stuff first. The UseHugeTLBFS stuff came later. See
> this discussion:
>
> http://mail.openjdk.java.net/pipermail/hotspot-dev/2011-April/004063.html
>
> and the related bug that made the changes:
>
> https://bugs.openjdk.java.net/browse/JDK-7034464
>
> HTH.
>
> David
>
> > I do not see a practical difference beside the fact that:
> > - System V shm API is slightly fussier to use (eg no partial unmap
> possible)
> > - it requires kernel.shmmax to be correctly set, in addition to the
> > vm.nr_xxx_hugepages
> > - it requires more user permissions
> >
> > But in the end, we end up with kernel huge pages, one way or the other.
> > Both give access to 2M and 1G pages.
> >
> > I looked through mailing list archives and bug descriptions, but beyond
> > vague mentionings of "one way could fail, then try the other" I did not
> > find anything.
> >
> > I must be missing something. Does anyone have any closer knowledge
> > about this?
> >
> > Thank you!
> >
> > Thomas
> >
>