RFR: 8353637: ZGC: Discontiguous memory reservation is broken on Windows

[Stefan Karlsson]

While working on [JDK-8350441](https://bugs.openjdk.org/browse/JDK-8350441) I realized that the current Windows implementation to use placeholders for reservations are broken if we ever fallback to using the part that performs discontiguous heap reservation.

To understand this bug you first need to understand how and why we use the placeholder mechanism. From zMapper_windows.cpp:


// Memory reservation, commit, views, and placeholders.
//
// To be able to up-front reserve address space for the heap views, and later
// multi-map the heap views to the same physical memory, without ever losing the
// reservation of the reserved address space, we use "placeholders".
//
// These placeholders block out the address space from being used by other parts
// of the process. To commit memory in this address space, the placeholder must
// be replaced by anonymous memory, or replaced by mapping a view against a
// paging file mapping. We use the later to support multi-mapping.
//
// We want to be able to dynamically commit and uncommit the physical memory of
// the heap (and also unmap ZPages), in granules of ZGranuleSize bytes. There is
// no way to grow and shrink the committed memory of a paging file mapping.
// Therefore, we create multiple granule-sized page file mappings. The memory is
// committed by creating a page file mapping, map a view against it, commit the
// memory, unmap the view. The memory will stay committed until all views are
// unmapped, and the paging file mapping handle is closed.
//
// When replacing a placeholder address space reservation with a mapped view
// against a paging file mapping, the virtual address space must exactly match
// an existing placeholder's address and size. Therefore we only deal with
// granule-sized placeholders at this layer. Higher layers that keep track of
// reserved available address space can (and will) coalesce placeholders, but
// they will be split before being used.

And the way we implement this is through the callbacks in zVirtualMemory_windows.cpp:


// Each reserved virtual memory address area registered in _manager is
// exactly covered by a single placeholder. Callbacks are installed so
// that whenever a memory area changes, the corresponding placeholder
// is adjusted.
//
// The create and grow callbacks are called when virtual memory is
// returned to the memory manager. The new memory area is then covered
// by a new single placeholder.
//
// The destroy and shrink callbacks are called when virtual memory is
// allocated from the memory manager. The memory area is then is split
// into granule-sized placeholders.
//
// See comment in zMapper_windows.cpp explaining why placeholders are
// split into ZGranuleSize sized placeholders.

So, we have the expectation that all memory areas in the memory manager should be covered by exactly one placeholder. We implement that by having the callbacks disabled while we initialize the reserved memory for the heap. This works as long as we get a contiguous memory reservation, and the code has various mechanisms to really try to get contiguous memory for the heap. However, if all those attempts fail, we have a fallback to reserve discontiguous memory. That mode uses interval halving to reserve exactly around the memory that is blocking use from getting a contiguous memory reservation. An example of this would be a request to reserve four "granules" (2MB), but the forth granule is already reserved:


+--A--+--B--+--C--+--D--+
                     ^ D is pre-reserved

After failing to reserve the four granules (A, B, C, D), the code will split the range into two halves (A, B) and (C, D), and try to reserve them individually. It will succeed to reserve (A, B) but not (C, D). So, the code registers (A, B) and proceeds to split (C, D) into two parts (C) and (D), and try to reserve them individually. It will succeed with (C) but fail with (D). So, the code registers (C). When (C) is registered, the code sees that (A, B) and (C) are adjacent and fuse them into one region (A, B, C). The problem is that we don't have any callbacks to also fuse the placeholders, so we are left with reservation placeholders over (A, B) and (C). Later one, when we want to use use (A, B) for the heap, the code works under the impression that we have on single placeholder over (A, B, C), so it tries to split that memory are into two placeholders (A, B) and (C). This fails with a fatal error, because Windows will refuse make this split since we already have split the placehold
 er.

The proposal to fix this is to first enable callbacks from the start, before the initializing memory reservation calls are made. And then to change the virtual memory manager to differentiate between the two kinds of insertion (and extraction) operations we have:

1) The first insert operation happens when we "registers" new virtual memory. This is what's done during initialization of ZGC.
2) The other insert operation happens when the system "hands back" memory to the virtual memory manager.

The reason why we need to separate these to is that in (1) the memory area has one placeholder that spans the provided memory area, but in (2) we the memory area has a placeholder for every 2MB granule (as described above).

So, the patch applies the 'insert' callback for the (2) areas to convert them into looking like (1), and then they both can use the same code to insert the memory into the virtual memory manager.

An opposite mechanism is used when "handing out" memory vs de-registering memory for being unreserved. Where the "handing out" operation will perform a 'remove' callback before actually handing out the memory, ensuring that the memory is covered by 2MB placeholders.

The shrink and grow operations are relieved of their previous duties to split and coalesce the 2MB placeholders and are now only tasked with splitting memory into two placeholders and combining two placeholders into one. This is handled by the 'grow' and 'shrink' callbacks.

To be able to provoke this bug we have written a small gtest, which I think has enough comments to explain what's going on and when things used to break down. The added tests uses enough high-level operations that I also had to add the support to unreserve memory, without it verification code starts to fail.

The added unreserve code also introduces calls to NMT to register the releasing of memory. This in turn is problematic because NMT doesn't support releasing a larger memory area than what we previously registered. So, we have a similar problem that we had with the placeholders that the code isn't prepared to have adjacent memory treated as a single memory area. This is going to be fixed by the on-going rewrites to NMT, but for now I've added a workaround to release memory in 2MB chunks, which is supported by the current NMT implementation.

I've moved some of the tests in test_zMapper_windows.cpp to the new test_zVirtualMemoryManager.cpp file so that we can run these tests on other platforms as well.

-------------

Commit messages:
 - 8353637: ZGC: Discontiguous memory reservation is broken on Windows

Changes: https://git.openjdk.org/jdk/pull/24443/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=24443&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8353637
  Stats: 813 lines in 17 files changed: 524 ins; 192 del; 97 mod
  Patch: https://git.openjdk.org/jdk/pull/24443.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/24443/head:pull/24443

PR: https://git.openjdk.org/jdk/pull/24443