RFR: 8253638: Cleanup os::reserve_memory and remove MAP_FIXED

[Thomas Stuefe]

On Fri, 25 Sep 2020 12:16:01 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> There's a code path through os::reserve_memory that uses MAP_FIXED. This path is used if a requested address is given.
>> It's very dangerous to use MAP_FIXED, since it causes pre-existing mappings to be replaced. Alternative, move safe,
>> functions exists; named os_attempt_reserve_memory_at.  It's not only dangerous to use that API, the Mac AARCH64 port
>> also can't use it in conjunction with its MAP_JIT.
>> I started to split os::reserve_memory into two functions; one that doesn't care about were the memory is mapped
>> (os::reserve_memory), and another that uses the dangerous MAP_FIXED flag (os::reserve_memory_at). However, I noticed
>> that it's only windows code that actually use os::reserve_memory with a requested address. All other usages have been
>> cleaned out and replaced with os::attempt_reserve_memory. And on windows os::attempt_reserve_memory uses
>> os::reserve_memory with comments that it's safe because it only "attempts" to place the mapping.  So, this patch: 1)
>> Removes the forcefully reserving of memory and its MAP_FIXED usages! 2) Changes the windows code to call the
>> attempt_reserve_memory_at version, that it actually did anyway. 3) Flips the call order between pd_reserve_memory and
>> pd_attempt_reserve_memory_at, on windows. 4) There's also some unification and split done to handle MEMFLAGS and fds.
>> This part needs followup cleanups (IMHO). 5) The AIX implementation already refused to use MAP_FIXED, and now it
>> doesn't need special code to handle that. 6) Small cleanups around anon_mmap and anon_mmap_aligned
>
> src/hotspot/os/bsd/os_bsd.cpp line 2026:
> 
>> 2024: static char* anon_mmap(char* requested_addr, size_t bytes) {
>> 2025:   int flags = MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS;
>> 2026:
> 
> Could you add a short oneliner comment here (and in front of the other mmap calls) that MAP_FIXED is explicitly left
> out?

Also, flags could possibly be const now (linux too). Leave that up to you.

-------------

PR: https://git.openjdk.java.net/jdk/pull/357