Supporting allocation of Java object heap on NV-DIMMS

Sat Jun 4 01:45:45 UTC 2016

Hello all,

I have an update to the patch,  adding support for Windows.
It uses the Windows APIs described in https://channel9.msdn.com/Events/Build/2016/P470 to map Java heap to NV-DIMM memory.

Link to webrev: http://cr.openjdk.java.net/~vdeshpande/8153327/webrev.02/
JBS entry : https://bugs.openjdk.java.net/browse/JDK-8153327

Will like to hear your feedback!!

Thanks
Kishor

> -----Original Message-----
> From: Kharbas, Kishor
> Sent: Tuesday, May 24, 2016 3:03 PM
> To: Per Liden; hotspot-gc-dev at openjdk.java.net
> Cc: Kharbas, Kishor
> Subject: RE: Supporting allocation of Java object heap on NV-DIMMS
> 
> Oops.. forgot to insert the link to the new webrev.
> 
> http://cr.openjdk.java.net/~vdeshpande/8153327/webrev.01/
> https://bugs.openjdk.java.net/browse/JDK-8153327
> 
> - Kishor
> 
> > -----Original Message-----
> > From: Kharbas, Kishor
> > Sent: Tuesday, May 24, 2016 2:39 PM
> > To: Per Liden; hotspot-gc-dev at openjdk.java.net
> > Cc: Kharbas, Kishor
> > Subject: RE: Supporting allocation of Java object heap on NV-DIMMS
> >
> > Sorry for the long hiatus on this thread.
> > As it was apparent that it would have been difficult to integrate the
> > patch  in JDK9, I took some time to re-implement the feature.
> >
> > In the new patch, I do not change the existing functions in 'os' class
> > which reserve memory for the heap.
> > The existing logic in ReservedSpace::initialize() and
> > ReservedHeapSpace::try_reserve_heap() to reserve a virtual address is
> > untouched. Only after a virtual address range is successfully
> > allocated, I call a newly added function which maps this address range
> > to physical memory on the NV-DIMM
> >
> > The new call is os::map_memory_to_file() and its implemented for Linux
> > at this point.
> > I wanted to know your thoughts before I implement the function for
> > windows and other OS'es.
> >
> > Thank you
> > Kishor
> >
> > > -----Original Message-----
> > > From: Kharbas, Kishor
> > > Sent: Wednesday, April 13, 2016 4:43 PM
> > > To: Per Liden; hotspot-gc-dev at openjdk.java.net
> > > Cc: Kharbas, Kishor
> > > Subject: RE: Supporting allocation of Java object heap on NV-DIMMS
> > >
> > > Hi Per,
> > >
> > > Here are the answers to your questions - 1. You have an interesting
> > > point about non-heap memory. The current prototype only allocates
> > > object heap on the new memory. We do not have plans to allocate
> > > other JVM structures like cardtables, bitmaps on the new memory
> > > device. In the future we will look into it.
> > >      For now we are very interested in getting this patch in JDK9
> > > atleast as an experimental feature.
> > >
> > > 2.  The 3D XPoint memory has the same semantics as DRAM, and no
> > > special handling is required from the user/JVM perspective.
> > > With regards to  pages sizes, ext4-DAX implicitly tries to allocate
> > > largest granularity pages as possible.
> > > The mmap() system call I use in the patch passes a 'fd' in the
> > > directory provided by the new cmd line option "FSDirForHeap". In
> > > this case, UseLargePages flag is ignored and the size of pages
> > > allocated is upto to the filesystem.
> > >
> > > Regards
> > > Kishor
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Per Liden [mailto:per.liden at oracle.com]
> > > > Sent: Wednesday, April 13, 2016 4:48 AM
> > > > To: Kharbas, Kishor; hotspot-gc-dev at openjdk.java.net
> > > > Subject: Re: Supporting allocation of Java object heap on NV-DIMMS
> > > >
> > > > Hi,
> > > >
> > > > On 2016-04-12 19:32, Kharbas, Kishor wrote:
> > > > > Hi!
> > > > >
> > > > > In addition to points made in previous email, we have seen
> > > > > performance in
> > > > application performance by having larger heaps on 3D XPoint. With
> > > > proper tuning, the upside of having less GCs outweighs the
> > > > negative impact of slower latency.
> > > > > Besides that, the patch is implemented so that it has least
> > > > > impact on
> > > > existing code path, to make sure there is no regression.
> > > > >
> > > > > Would like to hear your thoughts and also the possibility of
> > > > > getting it in JDK
> > > > 9.
> > > >
> > > > First of all, I can just note that JDK 9 Feature Complete date is
> > > > getting very close.
> > > >
> > > > There seems to be at least two layers here:
> > > >
> > > > 1) The command line flag to direct heap mappings to be backed by a
> file.
> > > >
> > > > Having a flag like this sounds overall reasonable and useful.
> > > >
> > > > What's your vision going forward here? Is there a reason to
> > > > believe we might want to use this of non-Java heap memory in some
> future?
> > > > Marking bitmaps, malloc(), etc? How this is implemented,
> > > > configured, options names, etc might be influenced if we foresee such
> a future.
> > > >
> > > > General testing of these code paths could be done by mapping a
> > > > file on tmpfs, or something similar.
> > > >
> > > >
> > > > 2) The specific use case where this flag is used to a map 3D
> > > > XPoint backed file.
> > > >
> > > > Will this memory have the same semantics as RAM, with regards to
> > > > ordering, atomics, etc? Will there be a need for white listing
> > > > known good types of memory/devices/filesystems which we can safely
> > > > use for backing the heap?
> > > >
> > > > How does this interact with various pages sizes? E.g. mapping 3D
> > > > XPoint memory with MAP_HUGETLB works?
> > > >
> > > > Testing of this obviously requires special hardware.
> > > >
> > > >
> > > > cheers,
> > > > Per
> > > >
> > > > >
> > > > > Please do let me know if I need to make any changes and improve
> > > > > the
> > > > patch in any way.
> > > > >
> > > > > Regards
> > > > > Kishor
> > > > >
> > > > > -----Original Message-----
> > > > > From: Kharbas, Kishor
> > > > > Sent: Tuesday, April 5, 2016 10:00 AM
> > > > > To: 'Per Liden'; hotspot-gc-dev at openjdk.java.net
> > > > > Cc: Kharbas, Kishor
> > > > > Subject: RE: Supporting allocation of Java object heap on
> > > > > NV-DIMMS
> > > > >
> > > > > Hi Per,
> > > > >
> > > > > We see application of this feature in large scale multi-JVM
> > > > > deployments to
> > > > increase resource utilization. We could run daemon/service JVMs
> > > > with heap on slower memory and thus free up DRAM for critical
> processes.
> > > > > At Intel we have done internal experiments on large middleware
> > > > applications and have seen benefit of this feature.
> > > > >
> > > > > Thanks
> > > > > Kishor
> > > > >
> > > > > -----Original Message-----
> > > > > From: Per Liden [mailto:per.liden at oracle.com]
> > > > > Sent: Monday, April 4, 2016 4:37 AM
> > > > > To: Kharbas, Kishor; hotspot-gc-dev at openjdk.java.net
> > > > > Subject: Re: Supporting allocation of Java object heap on
> > > > > NV-DIMMS
> > > > >
> > > > > Hi,
> > > > >
> > > > > On 2016-04-02 03:13, Kharbas, Kishor wrote:
> > > > >> Hello all,
> > > > >>
> > > > >> I would like to propose adding a capability in the JVM to
> > > > >> allocate object heap on new kinds of heterogeneous memories
> > > > >> which are
> > > > exposed
> > > > >> through a file system. This could be achieved by adding a
> > > > >> command line option to specify a path in the filesystem which
> > > > >> would be used for
> > > > >> mmap() system call.
> > > > >>
> > > > >> Details:
> > > > >>
> > > > >> Linux kernel 4.0 onwards supports DAX mode[1] in Ext4 file
> > > > >> system [2] which is designed to support new heterogeneous
> > > > >> memory devices such
> > > > as
> > > > >> Intel's 3D XPoint memory [3]. DAX mode removes the need for
> > > > >> page
> > > > cache
> > > > >> and provides a direct mapping of process virtual space to
> > > > >> storage
> > space.
> > > > >>
> > > > >> We would like to add a command line option called "FSDirForHeap"
> > > > >> which will - 1) Enable using a file system as backing store for
> > > > >> Java object heap 2) Use the provided path to create a temporary
> > > > >> file for mmap() system call.
> > > > >
> > > > > The new kinds of memories opens up for some new and interesting
> > > > possibilities. However, as long as DRAM is faster (which I assume
> > > > they will continue be for at least some time), I have a hard time
> > > > seeing that you want to place your whole heap onto the NV-DIMMs. I
> > > > can see that some applications might want to place some large
> > > > subset of their data
> > > there (e.g.
> > > > an in memory-database), but I'm also guessing that placing the
> > > > whole heap there would come with a noticeable impact on
> > > > performance? Any thoughts on that?
> > > > >
> > > > > Do you have any data to share with regards to performance
> > > > > difference on
> > > > some relevant workloads? That would help in getting a better
> > > > understanding of the feasibility of this mode of operation.
> > > > >
> > > > > cheers,
> > > > > Per
> > > > >
> > > > >>
> > > > >> The description of the feature can be found at
> > > > >> https://bugs.openjdk.java.net/browse/JDK-8153327
> > > > >>
> > > > >> and the webrev is uploaded at
> > > > >> http://cr.openjdk.java.net/~vdeshpande/8153327/webrev.00/
> > > > >>
> > > > >> References:
> > > > >>
> > > > >> [1]
> > > > >> https://www.kernel.org/doc/Documentation/filesystems/dax.txt
> > > > >>
> > > > >> [2]
> > > > >> https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
> > > > >>
> > > > >> [3]
> > > > >> https://newsroom.intel.com/press-kits/introducing-intel-optane-
> > > > >> te
> > > > >> ch nol
> > > > >> ogy-bringing-3d-xpoint-memory-to-storage-and-memory-products/
> > > > >>
> > > > >> Much appreciated!
> > > > >>
> > > > >> Thanks you.
> > > > >>
> > > > >> Kishor Kharbas
> > > > >>