RFR(s): 8077276: allocating heap with UseLargePages and HugeTLBFS may trash existing memory mappings (linux)

[David Holmes]

On 30/04/2015 4:59 AM, Thomas Stüfe wrote:
> Could I have another reviewer, please, and a sponsor?

Latest webrev is:

http://cr.openjdk.java.net/~stuefe/webrevs/8077276/webrev.05/webrev

Sorry not something I can review in depth.

David

> Thanks!
>
> Thomas
>
>
> On Thursday, April 9, 2015, Thomas Stüfe <thomas.stuefe at gmail.com> wrote:
>
>> Hi all,
>>
>> please review this fix to huge page allocation on Linux.
>>
>> bug: https://bugs.openjdk.java.net/browse/JDK-8077276
>> webrev:
>> http://cr.openjdk.java.net/~stuefe/webrevs/8077276/webrev.00/webrev/
>>
>> os::Linux::reserve_memory_special_huge_tlbfs_mixed() first establishes a
>> mapping with small pages over the whole requested range, then exchanges the
>> parts of the mapping which are aligned to large page size with large pages.
>>
>> Now, when establishing the first mapping, it uses os::reserve_memory()
>> with a potentially non-null req_addr. In that case, os::reserve_memory()
>> will do a mmap(MAP_FIXED).
>>
>> This will trash any pre-existing mappings.
>>
>> Note that I could not willingly reproduce the error with an unmodified VM.
>> But I added a reproduction case which demonstrated that if one were to call
>> os::Linux::reserve_memory_special_huge_tlbfs_mixed() with a non-NULL
>> req_addr, existing mappings would be trashed. Depending on where we are in
>> address space, we also would overwrite libc structures and crash
>> immediately.
>>
>> The repro case is part of the change, see changes in
>> test_reserve_memory_special_huge_tlbfs_mixed(), and can be executed with:
>>
>> ./images/jdk/bin/java -XX:+UseLargePages -XX:+UseHugeTLBFS -XX:-UseSHM
>> -XX:+ExecuteInternalVMTests -XX:+VerboseInternalVMTests
>>
>> The fix: instead of using os::reserve_memory(),
>> os::Linux::reserve_memory_special_huge_tlbfs_mixed() now calls mmap()
>> directly with the non-NULL req_addr, but without MAP_FIXED. This means the
>> OS may do its best to allocate at req_addr, but will not trash any
>> mappings. This also follows the pattern in
>> os::Linux::reserve_memory_special_huge_tlbfs_only(), which is a sister
>> function of os::Linux::reserve_memory_special_huge_tlbfs_mixed().
>>
>> Note also discussion at:
>> http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-April/017823.html
>> .
>>
>> Thanks for reviewing!
>>
>> Kind Regards, Thomas
>>
>>