Linux large pages: why do we have UseSHM?

[Thomas Stüfe]

Thanks for the tip Volker! I ended up cloning the repo, but it is good to
know that I don't have to.

Cheers, Thomas

On Sat, Dec 5, 2020 at 7:43 AM Volker Simonis <volker.simonis at gmail.com>
wrote:

> Thomas Stüfe <thomas.stuefe at gmail.com> schrieb am Sa., 5. Dez. 2020,
> 07:26:
>
>> Hi David,
>>
>> thanks a lot, David. This made a lot of things clearer (BTW I wish the
>> patch links would work for these old bugs in JBS).
>>
>
> You probably already know this, so just for reference. The patch hashes
> are still valid in all existing Mercurial repositories, just some of the
> old team repos have disappeared.
>
> The easiest and quickest way to view such a change is to click on the
> commit link and than replace the team repo with the corresponding main
> repo. E.g. instead of:
>
> http://hg.openjdk.java.net/jdk7/hotspot-rt/hotspot/rev/139667d9836a
>
> use:
>
> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/rev/139667d9836a
>
>
>
>> So here is what I understand:
>>
>> UseSHM had been there first.
>>
>> Then JDK-7034464 introduced Transparent Huge Pages. Those needs
>> madvise(MADV_HUGEPAGE). For TPH, allocating via mmap(MAP_HUGETLB) was
>> added
>> alongside the old shmget(). I am not sure why, would TPH not have worked
>> with shmget()? But anyway, for a while TLBFS was synonymous with TPH.
>>
>> Later we had https://bugs.openjdk.java.net/browse/JDK-8024838, which
>> claimed TPH were responsible for performance problems and switched support
>> for them off by default. This left the mmap(MAP_HUGETLB) allocation intact
>> but avoided now using TPH. So it gave us the new explicit-TLBFS mode
>> (UseHugeTLBFS).
>>
>> Interestingly, one of the reasons JDK-7034464 gave for using mmap()
>> instead
>> of SHM was that mmap can be committed on demand. That is true (you can
>> call
>> mmap on every part of a mapping), but later with
>> https://bugs.openjdk.java.net/browse/JDK-8007074 this capability had been
>> switched off. Commit on demand would need to mmap(MAP_HUGETLBFS) to commit
>> and uncommit a mapping, but that may fail if the huge page pool is
>> exhausted, which would make the mmap call fail, which would introduce a
>> hole in the reserved space which concurrently running code could
>> accidentally mmap into... so, in effect, TLBFS is always committed now.
>>
>> There are still some holes in my understanding. But mainly, I wonder
>> whether we still need UseSHM. Would love to hear opinions.
>>
>> Thanks! Thomas
>>
>> On Sat, Dec 5, 2020 at 6:33 AM David Holmes <david.holmes at oracle.com>
>> wrote:
>>
>> > Hi Thomas,
>> >
>> > On 4/12/2020 4:48 am, Thomas Stüfe wrote:
>> > > Hi,
>> > >
>> > > I wonder why we support two different ways (discounting THPs) to
>> allocate
>> > > huge pages on Linux.
>> > >
>> > > We support allocating huge pages both via System V shm APIs (UseSHM)
>> and
>> > > via the "newer" mmap Posix APIs (UseHugeTLBFS). The latter is default
>> for
>> > > +UseLargePages.
>> >
>> > We had the UseSHM stuff first. The UseHugeTLBFS stuff came later. See
>> > this discussion:
>> >
>> >
>> http://mail.openjdk.java.net/pipermail/hotspot-dev/2011-April/004063.html
>> >
>> > and the related bug that made the changes:
>> >
>> > https://bugs.openjdk.java.net/browse/JDK-7034464
>> >
>> > HTH.
>> >
>> > David
>> >
>> > > I do not see a practical difference beside the fact that:
>> > > - System V shm API is slightly fussier to use (eg no partial unmap
>> > possible)
>> > > - it requires kernel.shmmax to be correctly set, in addition to the
>> > > vm.nr_xxx_hugepages
>> > > - it requires more user permissions
>> > >
>> > > But in the end, we end up with kernel huge pages, one way or the
>> other.
>> > > Both give access to 2M and 1G pages.
>> > >
>> > > I looked through mailing list archives and bug descriptions, but
>> beyond
>> > > vague mentionings of "one way could fail, then try the other" I did
>> not
>> > > find anything.
>> > >
>> > > I must be missing something. Does anyone have any closer knowledge
>> > > about this?
>> > >
>> > > Thank you!
>> > >
>> > > Thomas
>> > >
>> >
>>
>