RFR(s): 8077276: allocating heap with UseLargePages and HugeTLBFS may trash existing memory mappings (linux)

[Thomas Stüfe]

Could I have another reviewer, please, and a sponsor?

Thanks!

Thomas


On Thursday, April 9, 2015, Thomas Stüfe <thomas.stuefe at gmail.com> wrote:

> Hi all,
>
> please review this fix to huge page allocation on Linux.
>
> bug: https://bugs.openjdk.java.net/browse/JDK-8077276
> webrev:
> http://cr.openjdk.java.net/~stuefe/webrevs/8077276/webrev.00/webrev/
>
> os::Linux::reserve_memory_special_huge_tlbfs_mixed() first establishes a
> mapping with small pages over the whole requested range, then exchanges the
> parts of the mapping which are aligned to large page size with large pages.
>
> Now, when establishing the first mapping, it uses os::reserve_memory()
> with a potentially non-null req_addr. In that case, os::reserve_memory()
> will do a mmap(MAP_FIXED).
>
> This will trash any pre-existing mappings.
>
> Note that I could not willingly reproduce the error with an unmodified VM.
> But I added a reproduction case which demonstrated that if one were to call
> os::Linux::reserve_memory_special_huge_tlbfs_mixed() with a non-NULL
> req_addr, existing mappings would be trashed. Depending on where we are in
> address space, we also would overwrite libc structures and crash
> immediately.
>
> The repro case is part of the change, see changes in
> test_reserve_memory_special_huge_tlbfs_mixed(), and can be executed with:
>
> ./images/jdk/bin/java -XX:+UseLargePages -XX:+UseHugeTLBFS -XX:-UseSHM
> -XX:+ExecuteInternalVMTests -XX:+VerboseInternalVMTests
>
> The fix: instead of using os::reserve_memory(),
> os::Linux::reserve_memory_special_huge_tlbfs_mixed() now calls mmap()
> directly with the non-NULL req_addr, but without MAP_FIXED. This means the
> OS may do its best to allocate at req_addr, but will not trash any
> mappings. This also follows the pattern in
> os::Linux::reserve_memory_special_huge_tlbfs_only(), which is a sister
> function of os::Linux::reserve_memory_special_huge_tlbfs_mixed().
>
> Note also discussion at:
> http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-April/017823.html
> .
>
> Thanks for reviewing!
>
> Kind Regards, Thomas
>
>