RFR(s): 8077276: allocating heap with UseLargePages and HugeTLBFS may trash existing memory mappings (linux)

[Thomas Stüfe]

Hi all,

I realize that this patch may look more complicated than it is.

Basically, the problem is that under certain conditions, memory is
allocated using mmap(addr, MAP_FIXED) for an initial reservation (e.g. java
heap), which may trash existing mappings.

The fix is basically just to remove the MAP_FIXED flag for the initial
allocation.

Fix looks more complicated than this because the test functions were
expanded to add regression tests for this fix.

Please tell me if I should dumb down this fix or add explanations.

Thanks & Kind regards, Thomas



On Thu, Apr 30, 2015 at 8:39 AM, David Holmes <david.holmes at oracle.com>
wrote:

> On 30/04/2015 4:59 AM, Thomas Stüfe wrote:
>
>> Could I have another reviewer, please, and a sponsor?
>>
>
> Latest webrev is:
>
> http://cr.openjdk.java.net/~stuefe/webrevs/8077276/webrev.05/webrev
>
> Sorry not something I can review in depth.
>
> David
>
>
>  Thanks!
>>
>> Thomas
>>
>>
>> On Thursday, April 9, 2015, Thomas Stüfe <thomas.stuefe at gmail.com> wrote:
>>
>>  Hi all,
>>>
>>> please review this fix to huge page allocation on Linux.
>>>
>>> bug: https://bugs.openjdk.java.net/browse/JDK-8077276
>>> webrev:
>>> http://cr.openjdk.java.net/~stuefe/webrevs/8077276/webrev.00/webrev/
>>>
>>> os::Linux::reserve_memory_special_huge_tlbfs_mixed() first establishes a
>>> mapping with small pages over the whole requested range, then exchanges
>>> the
>>> parts of the mapping which are aligned to large page size with large
>>> pages.
>>>
>>> Now, when establishing the first mapping, it uses os::reserve_memory()
>>> with a potentially non-null req_addr. In that case, os::reserve_memory()
>>> will do a mmap(MAP_FIXED).
>>>
>>> This will trash any pre-existing mappings.
>>>
>>> Note that I could not willingly reproduce the error with an unmodified
>>> VM.
>>> But I added a reproduction case which demonstrated that if one were to
>>> call
>>> os::Linux::reserve_memory_special_huge_tlbfs_mixed() with a non-NULL
>>> req_addr, existing mappings would be trashed. Depending on where we are
>>> in
>>> address space, we also would overwrite libc structures and crash
>>> immediately.
>>>
>>> The repro case is part of the change, see changes in
>>> test_reserve_memory_special_huge_tlbfs_mixed(), and can be executed with:
>>>
>>> ./images/jdk/bin/java -XX:+UseLargePages -XX:+UseHugeTLBFS -XX:-UseSHM
>>> -XX:+ExecuteInternalVMTests -XX:+VerboseInternalVMTests
>>>
>>> The fix: instead of using os::reserve_memory(),
>>> os::Linux::reserve_memory_special_huge_tlbfs_mixed() now calls mmap()
>>> directly with the non-NULL req_addr, but without MAP_FIXED. This means
>>> the
>>> OS may do its best to allocate at req_addr, but will not trash any
>>> mappings. This also follows the pattern in
>>> os::Linux::reserve_memory_special_huge_tlbfs_only(), which is a sister
>>> function of os::Linux::reserve_memory_special_huge_tlbfs_mixed().
>>>
>>> Note also discussion at:
>>> http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-April/017823.html
>>> .
>>>
>>> Thanks for reviewing!
>>>
>>> Kind Regards, Thomas
>>>
>>>
>>>