RFR: 8253638: Cleanup os::reserve_memory and remove MAP_FIXED

[Ioi Lam]

On Fri, 25 Sep 2020 11:24:17 GMT, Stefan Karlsson <stefank at openjdk.org> wrote:

> There's a code path through os::reserve_memory that uses MAP_FIXED. This path is used if a requested address is given.
> It's very dangerous to use MAP_FIXED, since it causes pre-existing mappings to be replaced. Alternative, move safe,
> functions exists; named os_attempt_reserve_memory_at.  It's not only dangerous to use that API, the Mac AARCH64 port
> also can't use it in conjunction with its MAP_JIT.
> I started to split os::reserve_memory into two functions; one that doesn't care about were the memory is mapped
> (os::reserve_memory), and another that uses the dangerous MAP_FIXED flag (os::reserve_memory_at). However, I noticed
> that it's only windows code that actually use os::reserve_memory with a requested address. All other usages have been
> cleaned out and replaced with os::attempt_reserve_memory. And on windows os::attempt_reserve_memory uses
> os::reserve_memory with comments that it's safe because it only "attempts" to place the mapping.  So, this patch: 1)
> Removes the forcefully reserving of memory and its MAP_FIXED usages! 2) Changes the windows code to call the
> attempt_reserve_memory_at version, that it actually did anyway. 3) Flips the call order between pd_reserve_memory and
> pd_attempt_reserve_memory_at, on windows. 4) There's also some unification and split done to handle MEMFLAGS and fds.
> This part needs followup cleanups (IMHO). 5) The AIX implementation already refused to use MAP_FIXED, and now it
> doesn't need special code to handle that. 6) Small cleanups around anon_mmap and anon_mmap_aligned

Looks good to me. I checked the existing calls and no one is calling os::reserve_memory with fixed==true, so this
change doesn't impact any existing functionality. It's good to remove a dangerous API.

-------------

Marked as reviewed by iklam (Reviewer).

PR: https://git.openjdk.java.net/jdk/pull/357