Fwd: RFR: 8233299: Implementation: JEP 365: ZGC on Windows

Fri Nov 22 15:37:52 UTC 2019

Thanks, Erik.

StefanK

On 2019-11-22 15:24, Erik Joelsson wrote:
> Build change looks good.
> 
> /Erik
> 
> On 2019-11-21 13:11, Stefan Karlsson wrote:
>> Hi,
>>
>> I'm looking for a review for this tiny build change:
>> https://cr.openjdk.java.net/~stefank/8233299/webrev.01/make/autoconf/hotspot.m4.udiff.html 
>>
>>
>> Thanks,
>> StefanK
>>
>> -------- Forwarded Message --------
>> Subject:     RFR: 8233299: Implementation: JEP 365: ZGC on Windows
>> Date:     Thu, 31 Oct 2019 11:18:20 +0100
>> From:     Stefan Karlsson <stefan.karlsson at oracle.com>
>> To:     hotspot-gc-dev <hotspot-gc-dev at openjdk.java.net>
>>
>>
>>
>> Hi all,
>>
>> Please review this patch to add ZGC support on Windows.
>>
>> https://cr.openjdk.java.net/~stefank/8233299/webrev.01/
>> https://bugs.openjdk.java.net/browse/JDK-8233299
>>
>> As mentioned in the JEP (https://openjdk.java.net/jeps/365), there 
>> were some preparation patches that needed to go in to pave the way for 
>> this patch:
>>
>> 8232601: ZGC: Parameterize the ZGranuleMap table size
>> 8232602: ZGC: Make ZGranuleMap ZAddress agnostic
>> 8232604: ZGC: Make ZVerifyViews mapping and unmapping precise
>> 8232648: ZGC: Move ATTRIBUTE_ALIGNED to the front of declarations
>> 8232649: ZGC: Add callbacks to ZMemoryManager
>> 8232650: ZGC: Add initialization hooks for OS specific code
>> 8232651: Add implementation of os::processor_id() for Windows
>>
>> ... they have all been pushed now.
>>
>> One important key-point to this implementation is to use the new 
>> Windows APIs that support reservation and mapping of memory through 
>> "placeholders": VirtualAlloc2, VirtualFreeEx, MapViewOfFile3, and 
>> UnmapViewOfFile2. These functions are available starting from version 
>> 1803 of Windows 10 and Windows Server. ZGC will lookup these symbols 
>> to determine if the Windows version supports these functions.
>>
>>
>> Correlating the text in the JEP with the code:
>>
>> * '"Support for multi-mapping memory". ZGC's use of colored pointers 
>> requires support for heap multi-mapping, so that the same physical 
>> memory can be accessed from multiple different locations in the 
>> process address space. On Windows, paging-file backed memory provides 
>> physical memory with an identity (a handle), which is unrelated to the 
>> virtual address where it is mapped. Using this identity allows ZGC to 
>> map the same physical memory into multiple locations.'
>>
>> We commit memory via paging file mappings and map views into that memory.
>>
>> The function ZMapper::create_and_commit_paging_file_mapping uses 
>> CreateFileMappingW with SEC_RESERVE to create this mapping, 
>> MapViewOfFile3 to map a temporary view into the mapping, VirtualAlloc2 
>> to commit the memory, and then UnmapViewOfFile2 to unmap the view.
>>
>> The reason to use SEC_RESERVE and the extra VirtualAlloc2, instead of 
>> SEC_COMMIT, is to ensure that the later multi-mappings of committed 
>> file mappings don't fail under low-memory situations. Earlier 
>> prototypes used SEC_COMMIT and saw these kind of OOME errors when 
>> mapping new views to already committed memory. The current 
>> platform-independent ZGC code isn't prepared to handle OOME errors 
>> when mapping views, so we chose this solution.
>>
>> MapViewOfFile3 is then used to multi-map into the committed memory.
>>
>> * '"Support for mapping paging-file backed memory into a reserved 
>> address space". The Windows memory management API is not as flexible 
>> as POSIX's mmap/munmap, especially when it comes to mapping file 
>> backed memory into a previously reserved address space region. To do 
>> this, ZGC will use the Windows concept of address space placeholders. 
>> The placeholder concept was introduced in version 1803 of Windows 10 
>> and Windows Server. ZGC support for older versions of Windows will not 
>> be implemented.'
>>
>> Before the placeholder APIs there was no way to first reserve a 
>> specific virtual memory range, and then map a view of a committed 
>> paging file over that range. The VirtuaAlloc function could be used to 
>> first reserve and then commit anonymous memory, but nothing similar 
>> existed for mapped views. Now with placeholders, we can create a 
>> placeholder reservation of memory with VirtualAlloc2, and then replace 
>> that reservation with MapViewOfFile3. When memory is unmapped, we can 
>> use UnmapViewOfFile2 to "preserve" the placeholder memory reservation.
>>
>>
>> * '"Support for mapping and unmapping arbitrary parts of the heap". 
>> ZGC's heap layout in combination with its dynamic sizing (and 
>> re-sizing) of heap pages requires support for mapping and unmapping 
>> arbitrary heap granules. This requirement in combination with Windows 
>> address space placeholders requires special attention, since 
>> placeholders must be explicitly split/coalesced by the program, as 
>> opposed to being automatically split/coalesced by the operating system 
>> (as on Linux).'
>>
>> Half of the preparation patches were put in place to support this. 
>> When replacing a placeholder with a view of the backing file, we need 
>> to exactly match the address and size of a placeholder. Also, when 
>> unmapping a view, we need to exactly match the address and size of the 
>> view, and replace it with a placeholder.
>>
>> To make it easier to map and unmap arbitrary parts of the heap, we 
>> split reserved memory into ZGranuleSize-sized placeholders. So, 
>> whenever we perform any of these operations, we know that any given 
>> memory range could be dealt with as a number of granules.
>>
>> When memory is reserved, but not mapped, it is registered in the 
>> ZVirtualMemoryManager. It splits memory into granule-sized placholders 
>> when reserved memory is fetched, and coalesces placeholders when 
>> reserved memory is handed back.
>>
>>
>> * '"Support for committing and uncommitting arbitrary parts of the 
>> heap". ZGC can commit and uncommit physical memory dynamically while 
>> the Java program is running. To support these operations the physical 
>> memory will be divided into, and backed by, multiple paging-file 
>> segments. Each paging-file segment corresponds to a ZGC heap granule, 
>> and can be committed and uncommitted independently of other segments.'
>>
>> Just like we can map and unmap in granules, we want to be able to 
>> commit and uncommit memory in granules. You can see how memory is 
>> committed and uncommitted in granules in 
>> ZBackingFile::commit_from_paging_file and 
>> ZBackingFile::uncommit_from_paging_file. Each committed granule is 
>> associated with one registered handle. When memory for a granule is 
>> uncommitted, the handle is closed. At this point, no views exist to 
>> the mapping and the memory is handed back to the OS.
>>
>>
>> Final point about ZPhysicalMemoryBacking. We've tried to make this 
>> file similar on all OSes, with the hope to be able to combine them 
>> when both the Windows and macOS ports have been merged.
>>
>> Thanks,
>> StefanK