Supporting allocation of Java object heap on NV-DIMMS

Tue May 24 21:39:05 UTC 2016

Sorry for the long hiatus on this thread.
As it was apparent that it would have been difficult to integrate the patch  in JDK9, I took some time to re-implement the feature.

In the new patch, I do not change the existing functions in 'os' class which reserve memory for the heap.
The existing logic in ReservedSpace::initialize() and ReservedHeapSpace::try_reserve_heap() to reserve a virtual address is untouched. Only after a virtual address range is successfully allocated, I call a newly added function which maps this address range to physical memory on the NV-DIMM

The new call is os::map_memory_to_file() and its implemented for Linux at this point.
I wanted to know your thoughts before I implement the function for windows and other OS'es.

Thank you
Kishor

> -----Original Message-----
> From: Kharbas, Kishor
> Sent: Wednesday, April 13, 2016 4:43 PM
> To: Per Liden; hotspot-gc-dev at openjdk.java.net
> Cc: Kharbas, Kishor
> Subject: RE: Supporting allocation of Java object heap on NV-DIMMS
> 
> Hi Per,
> 
> Here are the answers to your questions - 1. You have an interesting point
> about non-heap memory. The current prototype only allocates object heap
> on the new memory. We do not have plans to allocate other JVM structures
> like cardtables, bitmaps on the new memory device. In the future we will
> look into it.
>      For now we are very interested in getting this patch in JDK9 atleast as an
> experimental feature.
> 
> 2.  The 3D XPoint memory has the same semantics as DRAM, and no special
> handling is required from the user/JVM perspective.
> With regards to  pages sizes, ext4-DAX implicitly tries to allocate largest
> granularity pages as possible.
> The mmap() system call I use in the patch passes a 'fd' in the directory
> provided by the new cmd line option "FSDirForHeap". In this case,
> UseLargePages flag is ignored and the size of pages allocated is upto to the
> filesystem.
> 
> Regards
> Kishor
> 
> 
> 
> > -----Original Message-----
> > From: Per Liden [mailto:per.liden at oracle.com]
> > Sent: Wednesday, April 13, 2016 4:48 AM
> > To: Kharbas, Kishor; hotspot-gc-dev at openjdk.java.net
> > Subject: Re: Supporting allocation of Java object heap on NV-DIMMS
> >
> > Hi,
> >
> > On 2016-04-12 19:32, Kharbas, Kishor wrote:
> > > Hi!
> > >
> > > In addition to points made in previous email, we have seen
> > > performance in
> > application performance by having larger heaps on 3D XPoint. With
> > proper tuning, the upside of having less GCs outweighs the negative
> > impact of slower latency.
> > > Besides that, the patch is implemented so that it has least impact
> > > on
> > existing code path, to make sure there is no regression.
> > >
> > > Would like to hear your thoughts and also the possibility of getting
> > > it in JDK
> > 9.
> >
> > First of all, I can just note that JDK 9 Feature Complete date is
> > getting very close.
> >
> > There seems to be at least two layers here:
> >
> > 1) The command line flag to direct heap mappings to be backed by a file.
> >
> > Having a flag like this sounds overall reasonable and useful.
> >
> > What's your vision going forward here? Is there a reason to believe we
> > might want to use this of non-Java heap memory in some future? Marking
> > bitmaps, malloc(), etc? How this is implemented, configured, options
> > names, etc might be influenced if we foresee such a future.
> >
> > General testing of these code paths could be done by mapping a file on
> > tmpfs, or something similar.
> >
> >
> > 2) The specific use case where this flag is used to a map 3D XPoint
> > backed file.
> >
> > Will this memory have the same semantics as RAM, with regards to
> > ordering, atomics, etc? Will there be a need for white listing known
> > good types of memory/devices/filesystems which we can safely use for
> > backing the heap?
> >
> > How does this interact with various pages sizes? E.g. mapping 3D
> > XPoint memory with MAP_HUGETLB works?
> >
> > Testing of this obviously requires special hardware.
> >
> >
> > cheers,
> > Per
> >
> > >
> > > Please do let me know if I need to make any changes and improve the
> > patch in any way.
> > >
> > > Regards
> > > Kishor
> > >
> > > -----Original Message-----
> > > From: Kharbas, Kishor
> > > Sent: Tuesday, April 5, 2016 10:00 AM
> > > To: 'Per Liden'; hotspot-gc-dev at openjdk.java.net
> > > Cc: Kharbas, Kishor
> > > Subject: RE: Supporting allocation of Java object heap on NV-DIMMS
> > >
> > > Hi Per,
> > >
> > > We see application of this feature in large scale multi-JVM
> > > deployments to
> > increase resource utilization. We could run daemon/service JVMs with
> > heap on slower memory and thus free up DRAM for critical processes.
> > > At Intel we have done internal experiments on large middleware
> > applications and have seen benefit of this feature.
> > >
> > > Thanks
> > > Kishor
> > >
> > > -----Original Message-----
> > > From: Per Liden [mailto:per.liden at oracle.com]
> > > Sent: Monday, April 4, 2016 4:37 AM
> > > To: Kharbas, Kishor; hotspot-gc-dev at openjdk.java.net
> > > Subject: Re: Supporting allocation of Java object heap on NV-DIMMS
> > >
> > > Hi,
> > >
> > > On 2016-04-02 03:13, Kharbas, Kishor wrote:
> > >> Hello all,
> > >>
> > >> I would like to propose adding a capability in the JVM to allocate
> > >> object heap on new kinds of heterogeneous memories which are
> > exposed
> > >> through a file system. This could be achieved by adding a command
> > >> line option to specify a path in the filesystem which would be used
> > >> for
> > >> mmap() system call.
> > >>
> > >> Details:
> > >>
> > >> Linux kernel 4.0 onwards supports DAX mode[1] in Ext4 file system
> > >> [2] which is designed to support new heterogeneous memory devices
> > >> such
> > as
> > >> Intel's 3D XPoint memory [3]. DAX mode removes the need for page
> > cache
> > >> and provides a direct mapping of process virtual space to storage space.
> > >>
> > >> We would like to add a command line option called "FSDirForHeap"
> > >> which will - 1) Enable using a file system as backing store for
> > >> Java object heap 2) Use the provided path to create a temporary
> > >> file for mmap() system call.
> > >
> > > The new kinds of memories opens up for some new and interesting
> > possibilities. However, as long as DRAM is faster (which I assume they
> > will continue be for at least some time), I have a hard time seeing
> > that you want to place your whole heap onto the NV-DIMMs. I can see
> > that some applications might want to place some large subset of their data
> there (e.g.
> > an in memory-database), but I'm also guessing that placing the whole
> > heap there would come with a noticeable impact on performance? Any
> > thoughts on that?
> > >
> > > Do you have any data to share with regards to performance difference
> > > on
> > some relevant workloads? That would help in getting a better
> > understanding of the feasibility of this mode of operation.
> > >
> > > cheers,
> > > Per
> > >
> > >>
> > >> The description of the feature can be found at
> > >> https://bugs.openjdk.java.net/browse/JDK-8153327
> > >>
> > >> and the webrev is uploaded at
> > >> http://cr.openjdk.java.net/~vdeshpande/8153327/webrev.00/
> > >>
> > >> References:
> > >>
> > >> [1] https://www.kernel.org/doc/Documentation/filesystems/dax.txt
> > >>
> > >> [2] https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
> > >>
> > >> [3]
> > >> https://newsroom.intel.com/press-kits/introducing-intel-optane-tech
> > >> nol ogy-bringing-3d-xpoint-memory-to-storage-and-memory-products/
> > >>
> > >> Much appreciated!
> > >>
> > >> Thanks you.
> > >>
> > >> Kishor Kharbas
> > >>