Supporting allocation of Java object heap on NV-DIMMS

Tue May 24 22:03:23 UTC 2016

Oops.. forgot to insert the link to the new webrev.

http://cr.openjdk.java.net/~vdeshpande/8153327/webrev.01/
https://bugs.openjdk.java.net/browse/JDK-8153327

- Kishor

> -----Original Message-----
> From: Kharbas, Kishor
> Sent: Tuesday, May 24, 2016 2:39 PM
> To: Per Liden; hotspot-gc-dev at openjdk.java.net
> Cc: Kharbas, Kishor
> Subject: RE: Supporting allocation of Java object heap on NV-DIMMS
> 
> Sorry for the long hiatus on this thread.
> As it was apparent that it would have been difficult to integrate the patch  in
> JDK9, I took some time to re-implement the feature.
> 
> In the new patch, I do not change the existing functions in 'os' class which
> reserve memory for the heap.
> The existing logic in ReservedSpace::initialize() and
> ReservedHeapSpace::try_reserve_heap() to reserve a virtual address is
> untouched. Only after a virtual address range is successfully allocated, I call a
> newly added function which maps this address range to physical memory on
> the NV-DIMM
> 
> The new call is os::map_memory_to_file() and its implemented for Linux at
> this point.
> I wanted to know your thoughts before I implement the function for
> windows and other OS'es.
> 
> Thank you
> Kishor
> 
> > -----Original Message-----
> > From: Kharbas, Kishor
> > Sent: Wednesday, April 13, 2016 4:43 PM
> > To: Per Liden; hotspot-gc-dev at openjdk.java.net
> > Cc: Kharbas, Kishor
> > Subject: RE: Supporting allocation of Java object heap on NV-DIMMS
> >
> > Hi Per,
> >
> > Here are the answers to your questions - 1. You have an interesting
> > point about non-heap memory. The current prototype only allocates
> > object heap on the new memory. We do not have plans to allocate other
> > JVM structures like cardtables, bitmaps on the new memory device. In
> > the future we will look into it.
> >      For now we are very interested in getting this patch in JDK9
> > atleast as an experimental feature.
> >
> > 2.  The 3D XPoint memory has the same semantics as DRAM, and no
> > special handling is required from the user/JVM perspective.
> > With regards to  pages sizes, ext4-DAX implicitly tries to allocate
> > largest granularity pages as possible.
> > The mmap() system call I use in the patch passes a 'fd' in the
> > directory provided by the new cmd line option "FSDirForHeap". In this
> > case, UseLargePages flag is ignored and the size of pages allocated is
> > upto to the filesystem.
> >
> > Regards
> > Kishor
> >
> >
> >
> > > -----Original Message-----
> > > From: Per Liden [mailto:per.liden at oracle.com]
> > > Sent: Wednesday, April 13, 2016 4:48 AM
> > > To: Kharbas, Kishor; hotspot-gc-dev at openjdk.java.net
> > > Subject: Re: Supporting allocation of Java object heap on NV-DIMMS
> > >
> > > Hi,
> > >
> > > On 2016-04-12 19:32, Kharbas, Kishor wrote:
> > > > Hi!
> > > >
> > > > In addition to points made in previous email, we have seen
> > > > performance in
> > > application performance by having larger heaps on 3D XPoint. With
> > > proper tuning, the upside of having less GCs outweighs the negative
> > > impact of slower latency.
> > > > Besides that, the patch is implemented so that it has least impact
> > > > on
> > > existing code path, to make sure there is no regression.
> > > >
> > > > Would like to hear your thoughts and also the possibility of
> > > > getting it in JDK
> > > 9.
> > >
> > > First of all, I can just note that JDK 9 Feature Complete date is
> > > getting very close.
> > >
> > > There seems to be at least two layers here:
> > >
> > > 1) The command line flag to direct heap mappings to be backed by a file.
> > >
> > > Having a flag like this sounds overall reasonable and useful.
> > >
> > > What's your vision going forward here? Is there a reason to believe
> > > we might want to use this of non-Java heap memory in some future?
> > > Marking bitmaps, malloc(), etc? How this is implemented, configured,
> > > options names, etc might be influenced if we foresee such a future.
> > >
> > > General testing of these code paths could be done by mapping a file
> > > on tmpfs, or something similar.
> > >
> > >
> > > 2) The specific use case where this flag is used to a map 3D XPoint
> > > backed file.
> > >
> > > Will this memory have the same semantics as RAM, with regards to
> > > ordering, atomics, etc? Will there be a need for white listing known
> > > good types of memory/devices/filesystems which we can safely use for
> > > backing the heap?
> > >
> > > How does this interact with various pages sizes? E.g. mapping 3D
> > > XPoint memory with MAP_HUGETLB works?
> > >
> > > Testing of this obviously requires special hardware.
> > >
> > >
> > > cheers,
> > > Per
> > >
> > > >
> > > > Please do let me know if I need to make any changes and improve
> > > > the
> > > patch in any way.
> > > >
> > > > Regards
> > > > Kishor
> > > >
> > > > -----Original Message-----
> > > > From: Kharbas, Kishor
> > > > Sent: Tuesday, April 5, 2016 10:00 AM
> > > > To: 'Per Liden'; hotspot-gc-dev at openjdk.java.net
> > > > Cc: Kharbas, Kishor
> > > > Subject: RE: Supporting allocation of Java object heap on NV-DIMMS
> > > >
> > > > Hi Per,
> > > >
> > > > We see application of this feature in large scale multi-JVM
> > > > deployments to
> > > increase resource utilization. We could run daemon/service JVMs with
> > > heap on slower memory and thus free up DRAM for critical processes.
> > > > At Intel we have done internal experiments on large middleware
> > > applications and have seen benefit of this feature.
> > > >
> > > > Thanks
> > > > Kishor
> > > >
> > > > -----Original Message-----
> > > > From: Per Liden [mailto:per.liden at oracle.com]
> > > > Sent: Monday, April 4, 2016 4:37 AM
> > > > To: Kharbas, Kishor; hotspot-gc-dev at openjdk.java.net
> > > > Subject: Re: Supporting allocation of Java object heap on NV-DIMMS
> > > >
> > > > Hi,
> > > >
> > > > On 2016-04-02 03:13, Kharbas, Kishor wrote:
> > > >> Hello all,
> > > >>
> > > >> I would like to propose adding a capability in the JVM to
> > > >> allocate object heap on new kinds of heterogeneous memories which
> > > >> are
> > > exposed
> > > >> through a file system. This could be achieved by adding a command
> > > >> line option to specify a path in the filesystem which would be
> > > >> used for
> > > >> mmap() system call.
> > > >>
> > > >> Details:
> > > >>
> > > >> Linux kernel 4.0 onwards supports DAX mode[1] in Ext4 file system
> > > >> [2] which is designed to support new heterogeneous memory devices
> > > >> such
> > > as
> > > >> Intel's 3D XPoint memory [3]. DAX mode removes the need for page
> > > cache
> > > >> and provides a direct mapping of process virtual space to storage
> space.
> > > >>
> > > >> We would like to add a command line option called "FSDirForHeap"
> > > >> which will - 1) Enable using a file system as backing store for
> > > >> Java object heap 2) Use the provided path to create a temporary
> > > >> file for mmap() system call.
> > > >
> > > > The new kinds of memories opens up for some new and interesting
> > > possibilities. However, as long as DRAM is faster (which I assume
> > > they will continue be for at least some time), I have a hard time
> > > seeing that you want to place your whole heap onto the NV-DIMMs. I
> > > can see that some applications might want to place some large subset
> > > of their data
> > there (e.g.
> > > an in memory-database), but I'm also guessing that placing the whole
> > > heap there would come with a noticeable impact on performance? Any
> > > thoughts on that?
> > > >
> > > > Do you have any data to share with regards to performance
> > > > difference on
> > > some relevant workloads? That would help in getting a better
> > > understanding of the feasibility of this mode of operation.
> > > >
> > > > cheers,
> > > > Per
> > > >
> > > >>
> > > >> The description of the feature can be found at
> > > >> https://bugs.openjdk.java.net/browse/JDK-8153327
> > > >>
> > > >> and the webrev is uploaded at
> > > >> http://cr.openjdk.java.net/~vdeshpande/8153327/webrev.00/
> > > >>
> > > >> References:
> > > >>
> > > >> [1] https://www.kernel.org/doc/Documentation/filesystems/dax.txt
> > > >>
> > > >> [2] https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
> > > >>
> > > >> [3]
> > > >> https://newsroom.intel.com/press-kits/introducing-intel-optane-te
> > > >> ch nol
> > > >> ogy-bringing-3d-xpoint-memory-to-storage-and-memory-products/
> > > >>
> > > >> Much appreciated!
> > > >>
> > > >> Thanks you.
> > > >>
> > > >> Kishor Kharbas
> > > >>