MappedMemorySegment & in-memory filesystems

[Maurizio Cimadamore]

On 24/06/2020 07:57, Ty Young wrote:
>
> On 6/22/20 9:11 AM, Ty Young wrote:
>>
>>
>>>
>>> In other words, I think you should really go down the mapped segment 
>>> path if you care about persistence - if that's not your primary use 
>>> case, I think that's probably not the right tool for the job?
>>
>>
>> There is no other option - realloc, as you've said before, makes zero 
>> commitments in regards to whether the address you get back is purely 
>> just expanded. If I went down that path, I'd have to create a messy 
>> way to rebase all existing instances of MemoryAddress on the new 
>> memory chunk.
>>
>>
>> With MappedMemorySegment this is not the case as it's based on the 
>> same file. All existing instances of MemoryAddress are valid since 
>> they are backed by the file.
>>
>>
>
> If the answer is no then whatever. There are in-memory filesystems 
> available for Java, they just aren't apart of the JDK nor do they use 
> JPMS. Given that a large part of FMA deals with memory I thought it 
> was related.
We won't be looking into supporting in-file memory file system, at least 
not in the short term. As you say, there are options available there.
>
>
> Anyway, the fact that this works is kinda suspect. I wasn't able to 
> determine whether or not this is already known about internally after 
> gazing into my magical mind-reading crystal ball, so apologies I guess.
Honestly, I'm also surprised that you can map a memory file that comes 
from some exotic file system - but after asking around, it seems like 
this is how the file system API works - e.g. real file descriptors are 
created, which then make the mmap operation possible.
>
>
> TL;DR, by using MemorySegment.ofNativeRestricted you can create a 
> MemorySegment that is in a limbo state. This segment will not close 
> when closing the parent MemorySegment from which it came from. This 
> segment will always report that it is alive unless closed itself but 
> if the File backed by the MappedMemorySegment is closed it will crash 
> the JVM on read/write access. You can make it usable again by loading 
> the File backed by the previous MappedMemorySegment into memory again. 
> Of note is that the new MappedMemorySegment can be 1 byte even though, 
> for example, 8 bytes was carved out of the same file before.
ofNativeRestricted is a... restricted method, it's usage is unsafe, as 
clearly stated by the javadoc. When you create one of these, it's up to 
you to make sure that temporal bounds are managed correctly. There's 
absolutely no connection between "sliceSegment" and "unsafeSliceSegment".
>
>
> If this isn't known about already then it would be nice if it was made 
> part of the official API by adding an "isTethered()" method to 
> MappedMemorySegment. I'm not 100% what's going on internally here in 
> terms of memory state, my guess is that if the File backing a 
> MappedMemorySegment isn't in memory then no memory is consumed even 
> though the MemorySegment's return true for an alive state. The 
> documentation for isAlive doesn't define what being "alive" means.

isAlive means that a segment has not been closed - which in this case it 
hasn't. The ofNativeRestricted factory is an unsafe factory for creating 
views over existing memory region, where the user wants to explicitly 
manage temporal bounds. The way it's meant to be used is this:

* you get some random pointer from a native function (eg. some allocator)
* you wrap that pointer into a segment, with given size (and cleanup 
function, if you want)
* you discard the original pointer and start working only with the 
unsafe segment; when you close the unsafe segment, its cleanup action 
will be called (if one was provided)

You can of course use restricted segments for other purposes (such as 
get rid of confinement restrictions) but doing so require extra care 
because, as you have seen, you can easily shoot yourself in the foot 
(which is why you need a flag on the command line to activate this mode 
in the first place).

Maurizio


>
>
> Here is code to produce this:
>
>
> File file = new File("./test.txt");
>
>         if(!file.exists())
>             file.createNewFile();
>         else
>         {
>             file.delete();
>             file.createNewFile();
>
>         }
>
>         VarHandle handle = MemoryHandles.varHandle(long.class, 
> ByteOrder.nativeOrder());
>
>         MappedMemorySegment segment = 
> MemorySegment.mapFromPath(file.toPath(), 0, 128, 
> FileChannel.MapMode.READ_WRITE);
>
>         MemorySegment sliceSegment = segment.asSlice(0, 8);
>
>         handle.set(sliceSegment.baseAddress(), 20);
>
>         MemorySegment unsafeSliceSegment = 
> MemorySegment.ofNativeRestricted(
>                 sliceSegment.baseAddress(),
>                 CSupport.C_LONGLONG.byteSize(),
>                 null,
>                 null,
>                 null
>         );
>
>         segment.force();
>         segment.unload();
>         segment.close();
>
>         System.out.println("Big Segment Alive? " + segment.isAlive()); 
> // prints false
>         System.out.println("Sliced Segment Alive? " + 
> sliceSegment.isAlive()); // prints false
>         System.out.println("Unsafe Sliced Segment Alive? " + 
> unsafeSliceSegment.isAlive()); // prints true
>
>         // Removing this line gives a JVM crash on access of 
> unsafeSliceSegment
>         segment = MemorySegment.mapFromPath(file.toPath(), 0, 128, 
> FileChannel.MapMode.READ_WRITE);
>
> System.out.println(handle.get(unsafeSliceSegment.baseAddress()));
>