RFR: JDK-8266222: [aix] In mmap-mode, partial releases with os::release_memory may trash internal bookkeeping

[Thomas Stuefe]

In os_aix.cpp we keep book about for reserved address ranges since we need to keep information about which API (mmap or shmat) had been used for this range. (code hint: see vmembk_(add|remove|find)).

When releasing memory via os::release_memory, we remove those ranges from the internal bookkeeping.

However, for mmap() the release may be partial. In that case the internal bookkeeping is not updated correctly: we just remove the record for the whole original address range. 

After that operation, the remainder address range is still mapped from the point of the VM, but the bookkeeping is lost and subsequent calls to os::release_memory or os::commit/uncommit_memory for this range will run into a guarantee.

Partial releases can happen e.g. when, in the process of running os::reserve_memory_aligned(), the extra-aligned pages are unmapped. This is of more concern now with the new Metaspace, since we now reserve with larger alignments (4G+).

This problem was hidden by the fact that we typically run in 64K paged mode where we use SysV shared memory; but if AME is enabled on AIX, 64K pages are disabled, and the VM falls back to 4K pages and using mmap, and runs into this bug.

Edit: Tested on AIX machines with and without AME activated, solves the problems. All SAP nightlies ran, no problems caused by this patch.

-------------

Commit messages:
 - start

Changes: https://git.openjdk.java.net/jdk/pull/3765/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3765&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8266222
  Stats: 25 lines in 1 file changed: 19 ins; 1 del; 5 mod
  Patch: https://git.openjdk.java.net/jdk/pull/3765.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/3765/head:pull/3765

PR: https://git.openjdk.java.net/jdk/pull/3765