Supporting allocation of Java object heap on NV-DIMMS

[Per Liden]

Hi,

On 2016-04-12 19:32, Kharbas, Kishor wrote:
> Hi!
>
> In addition to points made in previous email, we have seen performance in application performance by having larger heaps on 3D XPoint. With proper tuning, the upside of having less GCs outweighs the negative impact of slower latency.
> Besides that, the patch is implemented so that it has least impact on existing code path, to make sure there is no regression.
>
> Would like to hear your thoughts and also the possibility of getting it in JDK 9.

First of all, I can just note that JDK 9 Feature Complete date is 
getting very close.

There seems to be at least two layers here:

1) The command line flag to direct heap mappings to be backed by a file.

Having a flag like this sounds overall reasonable and useful.

What's your vision going forward here? Is there a reason to believe we 
might want to use this of non-Java heap memory in some future? Marking 
bitmaps, malloc(), etc? How this is implemented, configured, options 
names, etc might be influenced if we foresee such a future.

General testing of these code paths could be done by mapping a file on 
tmpfs, or something similar.


2) The specific use case where this flag is used to a map 3D XPoint 
backed file.

Will this memory have the same semantics as RAM, with regards to 
ordering, atomics, etc? Will there be a need for white listing known 
good types of memory/devices/filesystems which we can safely use for 
backing the heap?

How does this interact with various pages sizes? E.g. mapping 3D XPoint 
memory with MAP_HUGETLB works?

Testing of this obviously requires special hardware.


cheers,
Per

>
> Please do let me know if I need to make any changes and improve the patch in any way.
>
> Regards
> Kishor
>
> -----Original Message-----
> From: Kharbas, Kishor
> Sent: Tuesday, April 5, 2016 10:00 AM
> To: 'Per Liden'; hotspot-gc-dev at openjdk.java.net
> Cc: Kharbas, Kishor
> Subject: RE: Supporting allocation of Java object heap on NV-DIMMS
>
> Hi Per,
>
> We see application of this feature in large scale multi-JVM deployments to increase resource utilization. We could run daemon/service JVMs with heap on slower memory and thus free up DRAM for critical processes.
> At Intel we have done internal experiments on large middleware applications and have seen benefit of this feature.
>
> Thanks
> Kishor
>
> -----Original Message-----
> From: Per Liden [mailto:per.liden at oracle.com]
> Sent: Monday, April 4, 2016 4:37 AM
> To: Kharbas, Kishor; hotspot-gc-dev at openjdk.java.net
> Subject: Re: Supporting allocation of Java object heap on NV-DIMMS
>
> Hi,
>
> On 2016-04-02 03:13, Kharbas, Kishor wrote:
>> Hello all,
>>
>> I would like to propose adding a capability in the JVM to allocate
>> object heap on new kinds of heterogeneous memories which are exposed
>> through a file system. This could be achieved by adding a command line
>> option to specify a path in the filesystem which would be used for
>> mmap() system call.
>>
>> Details:
>>
>> Linux kernel 4.0 onwards supports DAX mode[1] in Ext4 file system [2]
>> which is designed to support new heterogeneous memory devices such as
>> Intel's 3D XPoint memory [3]. DAX mode removes the need for page cache
>> and provides a direct mapping of process virtual space to storage space.
>>
>> We would like to add a command line option called "FSDirForHeap" which
>> will - 1) Enable using a file system as backing store for Java object
>> heap 2) Use the provided path to create a temporary file for mmap()
>> system call.
>
> The new kinds of memories opens up for some new and interesting possibilities. However, as long as DRAM is faster (which I assume they will continue be for at least some time), I have a hard time seeing that you want to place your whole heap onto the NV-DIMMs. I can see that some applications might want to place some large subset of their data there (e.g. an in memory-database), but I'm also guessing that placing the whole heap there would come with a noticeable impact on performance? Any thoughts on that?
>
> Do you have any data to share with regards to performance difference on some relevant workloads? That would help in getting a better understanding of the feasibility of this mode of operation.
>
> cheers,
> Per
>
>>
>> The description of the feature can be found at
>> https://bugs.openjdk.java.net/browse/JDK-8153327
>>
>> and the webrev is uploaded at
>> http://cr.openjdk.java.net/~vdeshpande/8153327/webrev.00/
>>
>> References:
>>
>> [1] https://www.kernel.org/doc/Documentation/filesystems/dax.txt
>>
>> [2] https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
>>
>> [3]
>> https://newsroom.intel.com/press-kits/introducing-intel-optane-technol
>> ogy-bringing-3d-xpoint-memory-to-storage-and-memory-products/
>>
>> Much appreciated!
>>
>> Thanks you.
>>
>> Kishor Kharbas
>>