FW: RFR(M): 8204908: Allocation of Old generation of Java Heap on alternate memory devices.

Wed Jun 13 18:24:51 UTC 2018

I did not have my subscription to runtime so it bounced.. just resending it to runtime...
Thanks,
Vinay

-----Original Message-----
From: hotspot-gc-dev [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of Awasthi, Vinay K
Sent: Wednesday, June 13, 2018 11:18 AM
To: Thomas Stüfe <thomas.stuefe at gmail.com>
Cc: Viswanathan, Sandhya <sandhya.viswanathan at intel.com>; hotspot-gc-dev at openjdk.java.net; Hotspot dev runtime <hotspot-runtime-dev at openjdk.java.net>; Kharbas, Kishor <kishor.kharbas at intel.com>
Subject: RE: RFR(M): 8204908: Allocation of Old generation of Java Heap on alternate memory devices.

First Thanks Thomas for your comments...

1. All changes are kept within AllocateOldGenAt variable being set enclosure... We do not want any changes to happen if this is NOT set.

On virtualspace.cpp
All changes are related to the fact that now Xmx gives overall memory to map, out of that part of the memory is mapped in for DRAM rest is reserved and later committed during expand--> calling make_region_available etc.. for G1GC. We no longer have one shot reservation/commit for entire space. We are only setting variables here (if _fd_for_nvdimm is set which is coming from AllocateOldGenAt...)... These setting are later used in g1PageBasedVirtualMemory (see os::nvdimm_heapbase() etc..). All changes again are kept out from leaking (except class variables in reservespace... 

On ReserveSpace:
2. Changes in ReservedSpace are related to following:
	1. We want to reserve enough virtual memory so that later there is room to map NVDIMM.
		a. We envision configurations where 1 TB NVDIMM is being used within a system with 32-64 GB memory. User will provide -Xmx and G1MaxNewSizePercent (this will limit DRAM allocation of YoungGen) and rest will be mapped to NVDIMM.. currently ERGO is not aware of 2 kinds of memories present in the system with different performance characteristics (read/write latencies etc...)... I did not want to make any other changes any where else... 
		b. We setup nvdimm_base memory and sizes etc.. for NVDIMM and commit it when expand is called in g1PageBasedVirtualMemory.
		c. All committing is happening with in map_memory_to_file which is called when AllocateHeapAt or AllocateOldGenAt is used. Rest of the VM uses in case of Linux etc.. os::Linux:commit_memory_impl which I did not want to modify to keep changes local (as we need MAP_SHARED in case of file descriptors..)

On os::commit_memory()
3. Since I did not want to have *ANY* change to become active if AllocateOldGenAt is not used... I added commit_memory variance that takes file descriptor and offset also as parameter.. This is *ONLY* called when AllocateOldGenAt is used by OldGen to commit memory incrementally as we got feedback earlier that JVM team will like to have support for UseAdaptiveSizePolicy as this resizes OldGen (in our case NVDIMM mapping) and YoungGen (using DRAM) as needed. We see about 20-28% gain in performance using tmpfs for Specjbb2005 (vs fixing OldGen to a fixed size and mapping it all in one shot.... On Windows, there is NO way to do incremental mapping so we are committing oldGen to maximum allowed from get go)...

JVM is continuing to use commit_memory without file descriptors & offset in all cases if AllocateOldGenAt is not used. If AllocateOldGenAt is used commit_meory is ONLY called by OldGen to map using map_memory_to_file (again stand alone function that rest of the VM does not use) to do incremental mapping.

_fd_for_heap was added to map whole heap on NVDIMM. 

These are minimum set of changes that we need to support NVDIMM (i.e. we need to open a file, map it and commit)... Other changes are related to option handling etc.. i.e. do not allow AllocateOldGenAt if not G1GC and POGC... 

With this context in place, I am looking for any and all feedback...

Thanks,
Vinay

Regarding Reserve Space:
This implementation is using map_memory_to_file in all cases.. which is kept separate

-----Original Message-----
From: Thomas Stüfe [mailto:thomas.stuefe at gmail.com]
Sent: Tuesday, June 12, 2018 11:11 PM
To: Awasthi, Vinay K <vinay.k.awasthi at intel.com>
Cc: hotspot-gc-dev at openjdk.java.net; Kharbas, Kishor <kishor.kharbas at intel.com>; Viswanathan, Sandhya <sandhya.viswanathan at intel.com>; Hotspot dev runtime <hotspot-runtime-dev at openjdk.java.net>
Subject: Re: RFR(M): 8204908: Allocation of Old generation of Java Heap on alternate memory devices.

Hi,

(adding hs-runtime)

had a cursory glance at the proposed changes. I am taken aback by the amount of complexity added to ReservedSpace for what I understand is a narrow experimental feature only benefiting 1-2 Operating Systems and
- I guess, the JEP is not really clear there - only x86 with certain hardware configurations?

e.g. http://cr.openjdk.java.net/~kkharbas/8202286/webrev.00/share/memory/virtualspace.hpp.udiff.html

The source having zero comments does not really help either.

"The motivation behind this JEP is to provide an experimental feature to facilitate exploration of different use cases for these non-DRAM memories."

Ok, but does this really have to be upstream, in this form, to experiment with it?

I am not objecting against this feature in general. But I am unhappy about the monster ReservedSpace is turning into. IMHO before increasing complexity even further this should be revamped, otherwise it becomes too unwieldy to do anything with it. It already somehow takes care of a number of huge pages ("_special" and "_alignment"), implicit-null-checks-when-space-happens-to-be-used-as-part-of-java-heap
("_noaccess_prefix"), allocation at alternate file locations ("_fd_for_heap", introduced by you with 8190308).

You also added a new variant to os::commit_memory() which directly goes down to mmap(). So far, for the most part, the os::{reserve|commit}_memory APIs have been agnostic to the underlying implementation. You pretty much tie it to mmap() now. This adds implicit restrictions to the API we did not have before (e.g. will not work if platform uses SysV shm APIs to implement these APIs).

Best Regards, Thomas

On Tue, Jun 12, 2018 at 9:32 PM, Awasthi, Vinay K <vinay.k.awasthi at intel.com> wrote:
> Hello,
>
> I am requesting comments on POGC/G1GC supporting NVDIMM/DRAM heaps. 
> When user supplies AllocateOldGenAt=<NVDIMM file path>, JVM divides 
> heap into 2 parts. First part is on NVDIMM where long living objects 
> go (OldGen) and other part is on DRAM where short living objects 
> reside(YoungGen). This is ONLY supported for G1GC and POGC collectors on Linux and Windows.
>
> On Windows, OldGen resizing is NOT supported. On Linux, for G1GC, 
> OldGen resizing is not supported however for POGC it is. Heap residing 
> on DRAM is supported for Windows and Linux for POGC and G1GC.
>
> JEP to support allocating Old generation on NV-DIMM -
> https://bugs.openjdk.java.net/browse/JDK-8202286
>
> Patch is at http://cr.openjdk.java.net/~kkharbas/8202286/webrev.00/
>
> SpecJbb2005/SpecJbb2015 etc. are passing with this patch and one can 
> test this by simply mounting tmpfs of certain size and pass that as an 
> argument to AllocateOldGenAt.
>
> For G1GC, G1MaxNewSizePercent controls how much of total heap will 
> reside on DRAM. Rest of the heap then goes to NVDIMM.
>
> For POGC, MaxNewSize decides the DRAM residing young gen size. Rest is 
> mounted on NVDIMM.
>
> In all these implementations, JVM ends up reserving more than initial 
> size determined by ergonomics (never more than Xmx). JVM displays 
> these messages and shows NVDIMM and DRAM reserved bytes.
>
> Thanks,
>
> Vinay
>
>
>
>
>
>