MemorySegment JVM memory leak

[Maurizio Cimadamore]

Hi,
I've tried your example and I think it's running as expected. With a 
little caveat (described below).

On my machine, the test completes  - with resident memory pegged at 
about 8G, while virtual memory was 100G. The latter is normal, since in 
order to make the memory accessible to your process, mmap has to reserve 
100G memory in the virtual address space. These 100G will not of course 
be committed all at once - the policy by which this is done is heavily 
OS dependent. Most OS will have some logic in order to discard unused 
pages, so that your application will not crash; also, most OS will also 
attempt to "prefetch" more than one page in order to speed up access.

Now, what's puzzling is why the resident memory is so high - and I think 
I found out what happens: basically this test is generating an awful lot 
of dirty pages - since these pages are not flushed back to disk (e.g. in 
a way similar to what MappedByteBuffer::force does), all these pages 
have to be kept around for longer in main memory (again, the details and 
thresholds are system specific).

Since I'm in the middle of adding force() support to mapped segments:

https://git.openjdk.java.net/panama-foreign/pull/115

I did a simple experiment: I added a call to the new 
MappedMemorySegment::force() after the call to MemoryAddress::copy, and 
re-ran the test. And now the resident memory was pegged at 150KB :-)

So, I believe your issue is that, when managing very large file you have 
to be disciplined in how you sync contents of main memory back into the 
mapped file - if you leave it implicit (e.g. to the OS), you might end 
up in a not-so-desirable place.

Does this help?

Maurizio

On 18/04/2020 17:33, ahmdprog.java at gmail.com wrote:
> Gentlemen,
>
>   
>
> There is memory leak in JVM while writing and reading byte array to memory
> segment. The below is simple example that generates 100G file with zero
> bytes. While running the below code, you will see that JVM consumes all
> server memory.
>
>   
>
> Unfortunately, I tested also in reading array of bytes. It has same issue of
> memory leak.
>
>   
>
>   
>
> public static void testingMemorySegment() {
>
>          String strFileName = "/disk3/bigdata.index" +
> System.currentTimeMillis();
>
>          File fileObjectFileName = new File(strFileName);
>
>          if (fileObjectFileName.exists() == false) {
>
>              try {
>
>                  fileObjectFileName.createNewFile();
>
>              } catch (IOException e) {
>
>              } catch (Exception e) {
>
>              }
>
>          }
>
>          long lngMemorySegmentFileSize = 107374182400l; // 100 G
>
>          byte[] bytesArrayString = new byte[4096];
>
>          MemorySegment sourceSegment =
> MemorySegment.ofArray(bytesArrayString);
>
>          long lngTotalNumberOfPagesForAllFile = lngMemorySegmentFileSize /
> 4096;
>
>          try {
>
>              MemorySegment memorySegmentTmp = MemorySegment.mapFromPath(new
> File(strFileName).toPath(), lngMemorySegmentFileSize,
> FileChannel.MapMode.READ_WRITE);
>
>              MemoryAddress address = memorySegmentTmp.baseAddress();
>
>              MemoryAddress sourceAddress = sourceSegment.baseAddress();
>
>              for (long i = 0; i < lngTotalNumberOfPagesForAllFile; i++) {
>
>                  MemoryAddress.copy(sourceAddress, address.addOffset(i *
> 4096), 4096);
>
>              }
>
>              memorySegmentTmp.close();
>
>          } catch (IOException e) {
>
>              e.printStackTrace();
>
>          }
>
>      }
>