RFR: 8253638: Cleanup os::reserve_memory and remove MAP_FIXED

[Stefan Karlsson]

On Fri, 25 Sep 2020 19:00:56 GMT, Ioi Lam <iklam at openjdk.org> wrote:

>> There's a code path through os::reserve_memory that uses MAP_FIXED. This path is used if a requested address is given.
>> It's very dangerous to use MAP_FIXED, since it causes pre-existing mappings to be replaced. Alternative, move safe,
>> functions exists; named os_attempt_reserve_memory_at.  It's not only dangerous to use that API, the Mac AARCH64 port
>> also can't use it in conjunction with its MAP_JIT.
>> I started to split os::reserve_memory into two functions; one that doesn't care about were the memory is mapped
>> (os::reserve_memory), and another that uses the dangerous MAP_FIXED flag (os::reserve_memory_at). However, I noticed
>> that it's only windows code that actually use os::reserve_memory with a requested address. All other usages have been
>> cleaned out and replaced with os::attempt_reserve_memory. And on windows os::attempt_reserve_memory uses
>> os::reserve_memory with comments that it's safe because it only "attempts" to place the mapping.  So, this patch: 1)
>> Removes the forcefully reserving of memory and its MAP_FIXED usages! 2) Changes the windows code to call the
>> attempt_reserve_memory_at version, that it actually did anyway. 3) Flips the call order between pd_reserve_memory and
>> pd_attempt_reserve_memory_at, on windows. 4) There's also some unification and split done to handle MEMFLAGS and fds.
>> This part needs followup cleanups (IMHO). 5) The AIX implementation already refused to use MAP_FIXED, and now it
>> doesn't need special code to handle that. 6) Small cleanups around anon_mmap and anon_mmap_aligned
>
> Looks good to me. I checked the existing calls and no one is calling os::reserve_memory with fixed==true, so this
> change doesn't impact any existing functionality. It's good to remove a dangerous API.

Thanks for reviewing!

-------------

PR: https://git.openjdk.java.net/jdk/pull/357