Supporting allocation of Java object heap on NV-DIMMS

Wed Apr 13 20:36:24 UTC 2016

Am Wed, 13 Apr 2016 13:47:53 +0200
schrieb Per Liden <per.liden at oracle.com>:

> Hi,
> 
> On 2016-04-12 19:32, Kharbas, Kishor wrote:
> > Hi!
> >
> > In addition to points made in previous email, we have seen
> > performance in application performance by having larger heaps on 3D
> > XPoint. With proper tuning, the upside of having less GCs outweighs
> > the negative impact of slower latency. Besides that, the patch is
> > implemented so that it has least impact on existing code path, to
> > make sure there is no regression.
> >
> > Would like to hear your thoughts and also the possibility of
> > getting it in JDK 9.
> 
> First of all, I can just note that JDK 9 Feature Complete date is 
> getting very close.
> 
> There seems to be at least two layers here:
> 
> 1) The command line flag to direct heap mappings to be backed by a
> file.
> 
> Having a flag like this sounds overall reasonable and useful.

On the other hand, without a proper GC for large heap the utility in
that is limited.

> What's your vision going forward here? Is there a reason to believe
> we might want to use this of non-Java heap memory in some future?
> Marking bitmaps, malloc(), etc? How this is implemented, configured,
> options names, etc might be influenced if we foresee such a future.

We do have however already the direct buffers, thats why I do think
this would be a nice fit. Some datagrid/cache systems would want to do
their own management, you only need to give them a reasonable amount of
backing pages.

> General testing of these code paths could be done by mapping a file
> on tmpfs, or something similar.
> 
> 
> 2) The specific use case where this flag is used to a map 3D XPoint 
> backed file.
> 
> Will this memory have the same semantics as RAM, with regards to 
> ordering, atomics, etc? Will there be a need for white listing known 
> good types of memory/devices/filesystems which we can safely use for 
> backing the heap?
> 
> How does this interact with various pages sizes? E.g. mapping 3D
> XPoint memory with MAP_HUGETLB works?
> 
> Testing of this obviously requires special hardware.
> 
> 
> cheers,
> Per
> 
> >
> > Please do let me know if I need to make any changes and improve the
> > patch in any way.
> >
> > Regards
> > Kishor
> >
> > -----Original Message-----
> > From: Kharbas, Kishor
> > Sent: Tuesday, April 5, 2016 10:00 AM
> > To: 'Per Liden'; hotspot-gc-dev at openjdk.java.net
> > Cc: Kharbas, Kishor
> > Subject: RE: Supporting allocation of Java object heap on NV-DIMMS
> >
> > Hi Per,
> >
> > We see application of this feature in large scale multi-JVM
> > deployments to increase resource utilization. We could run
> > daemon/service JVMs with heap on slower memory and thus free up
> > DRAM for critical processes. At Intel we have done internal
> > experiments on large middleware applications and have seen benefit
> > of this feature.
> >
> > Thanks
> > Kishor
> >
> > -----Original Message-----
> > From: Per Liden [mailto:per.liden at oracle.com]
> > Sent: Monday, April 4, 2016 4:37 AM
> > To: Kharbas, Kishor; hotspot-gc-dev at openjdk.java.net
> > Subject: Re: Supporting allocation of Java object heap on NV-DIMMS
> >
> > Hi,
> >
> > On 2016-04-02 03:13, Kharbas, Kishor wrote:
> >> Hello all,
> >>
> >> I would like to propose adding a capability in the JVM to allocate
> >> object heap on new kinds of heterogeneous memories which are
> >> exposed through a file system. This could be achieved by adding a
> >> command line option to specify a path in the filesystem which
> >> would be used for mmap() system call.
> >>
> >> Details:
> >>
> >> Linux kernel 4.0 onwards supports DAX mode[1] in Ext4 file system
> >> [2] which is designed to support new heterogeneous memory devices
> >> such as Intel's 3D XPoint memory [3]. DAX mode removes the need
> >> for page cache and provides a direct mapping of process virtual
> >> space to storage space.
> >>
> >> We would like to add a command line option called "FSDirForHeap"
> >> which will - 1) Enable using a file system as backing store for
> >> Java object heap 2) Use the provided path to create a temporary
> >> file for mmap() system call.
> >
> > The new kinds of memories opens up for some new and interesting
> > possibilities. However, as long as DRAM is faster (which I assume
> > they will continue be for at least some time), I have a hard time
> > seeing that you want to place your whole heap onto the NV-DIMMs. I
> > can see that some applications might want to place some large
> > subset of their data there (e.g. an in memory-database), but I'm
> > also guessing that placing the whole heap there would come with a
> > noticeable impact on performance? Any thoughts on that?
> >
> > Do you have any data to share with regards to performance
> > difference on some relevant workloads? That would help in getting a
> > better understanding of the feasibility of this mode of operation.
> >
> > cheers,
> > Per
> >
> >>
> >> The description of the feature can be found at
> >> https://bugs.openjdk.java.net/browse/JDK-8153327
> >>
> >> and the webrev is uploaded at
> >> http://cr.openjdk.java.net/~vdeshpande/8153327/webrev.00/
> >>
> >> References:
> >>
> >> [1] https://www.kernel.org/doc/Documentation/filesystems/dax.txt
> >>
> >> [2] https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
> >>
> >> [3]
> >> https://newsroom.intel.com/press-kits/introducing-intel-optane-technol
> >> ogy-bringing-3d-xpoint-memory-to-storage-and-memory-products/
> >>
> >> Much appreciated!
> >>
> >> Thanks you.
> >>
> >> Kishor Kharbas
> >>