MappedMemorySegment & in-memory filesystems

[Ty Young]

On 6/24/20 4:40 AM, Maurizio Cimadamore wrote:
>
> On 24/06/2020 07:57, Ty Young wrote:
>>
>> On 6/22/20 9:11 AM, Ty Young wrote:
>>>
>>>
>>>>
>>>> In other words, I think you should really go down the mapped 
>>>> segment path if you care about persistence - if that's not your 
>>>> primary use case, I think that's probably not the right tool for 
>>>> the job?
>>>
>>>
>>> There is no other option - realloc, as you've said before, makes 
>>> zero commitments in regards to whether the address you get back is 
>>> purely just expanded. If I went down that path, I'd have to create a 
>>> messy way to rebase all existing instances of MemoryAddress on the 
>>> new memory chunk.
>>>
>>>
>>> With MappedMemorySegment this is not the case as it's based on the 
>>> same file. All existing instances of MemoryAddress are valid since 
>>> they are backed by the file.
>>>
>>>
>>
>> If the answer is no then whatever. There are in-memory filesystems 
>> available for Java, they just aren't apart of the JDK nor do they use 
>> JPMS. Given that a large part of FMA deals with memory I thought it 
>> was related.
> We won't be looking into supporting in-file memory file system, at 
> least not in the short term. As you say, there are options available 
> there.
>>
>>
>> Anyway, the fact that this works is kinda suspect. I wasn't able to 
>> determine whether or not this is already known about internally after 
>> gazing into my magical mind-reading crystal ball, so apologies I guess.
> Honestly, I'm also surprised that you can map a memory file that comes 
> from some exotic file system - but after asking around, it seems like 
> this is how the file system API works - e.g. real file descriptors are 
> created, which then make the mmap operation possible.


I wasn't referring to that. What I'm calling suspect was FMA's API for 
it. According to Wikipedia[1]:


I/O errors on the underlying file (e.g. its removable drive is unplugged 
or optical media is ejected, disk full when writing, etc.) while 
accessing its mapped memory are reported to the application as the 
SIGSEGV/SIGBUS signals on POSIX, and the EXECUTE_IN_PAGE_ERROR 
structured exception on Windows. All code accessing mapped memory must 
be prepared to handle these errors, which don't normally occur when 
accessing memory.


In other words, if I delete the file or it becomes unreachable, the 
application should crash from my understanding but it doesn't. I don't 
know enough about the technical details so that's why I'm asking what 
the relationship here is.


>>
>>
>> TL;DR, by using MemorySegment.ofNativeRestricted you can create a 
>> MemorySegment that is in a limbo state. This segment will not close 
>> when closing the parent MemorySegment from which it came from. This 
>> segment will always report that it is alive unless closed itself but 
>> if the File backed by the MappedMemorySegment is closed it will crash 
>> the JVM on read/write access. You can make it usable again by loading 
>> the File backed by the previous MappedMemorySegment into memory 
>> again. Of note is that the new MappedMemorySegment can be 1 byte even 
>> though, for example, 8 bytes was carved out of the same file before.
> ofNativeRestricted is a... restricted method, it's usage is unsafe, as 
> clearly stated by the javadoc. When you create one of these, it's up 
> to you to make sure that temporal bounds are managed correctly. 
> There's absolutely no connection between "sliceSegment" and 
> "unsafeSliceSegment".


I'm well aware of the dangers of ofNativeRestricted. It's been brought 
up enough times that it's beyond a dead horse. I guess I just didn't 
realize it could be used to bring MemorySegments back from the dead, in 
a way.


>>
>>
>> If this isn't known about already then it would be nice if it was 
>> made part of the official API by adding an "isTethered()" method to 
>> MappedMemorySegment. I'm not 100% what's going on internally here in 
>> terms of memory state, my guess is that if the File backing a 
>> MappedMemorySegment isn't in memory then no memory is consumed even 
>> though the MemorySegment's return true for an alive state. The 
>> documentation for isAlive doesn't define what being "alive" means.
>
> isAlive means that a segment has not been closed - which in this case 
> it hasn't. The ofNativeRestricted factory is an unsafe factory for 
> creating views over existing memory region, where the user wants to 
> explicitly manage temporal bounds. The way it's meant to be used is this:
>
> * you get some random pointer from a native function (eg. some allocator)
> * you wrap that pointer into a segment, with given size (and cleanup 
> function, if you want)
> * you discard the original pointer and start working only with the 
> unsafe segment; when you close the unsafe segment, its cleanup action 
> will be called (if one was provided)
>
> You can of course use restricted segments for other purposes (such as 
> get rid of confinement restrictions) but doing so require extra care 
> because, as you have seen, you can easily shoot yourself in the foot 
> (which is why you need a flag on the command line to activate this 
> mode in the first place).


I'm aware of all this in the context of normal MemorySegments. 
MappedMemorySegments have an additional resource that is critical to how 
they work and that's not being factored into isAlive(), which is what I 
was getting at. If the file is missing that is being mapped, should it 
be considered alive? Or should there be a completely different concept 
of "tethering" which means the underling file is reachable and available?


Just clarifying what I meant. I'm reading the Wikipedia article and 
other resources and what FMA is doing isn't matching up with what I'm 
reading nor does it expose information that seems relevant. Maybe I'm 
not looking at the right information or misunderstanding something. 
Wouldn't be the first time.


[1] https://en.wikipedia.org/wiki/Memory-mapped_file


>
> Maurizio
>
>
>>
>>
>> Here is code to produce this:
>>
>>
>> File file = new File("./test.txt");
>>
>>         if(!file.exists())
>>             file.createNewFile();
>>         else
>>         {
>>             file.delete();
>>             file.createNewFile();
>>
>>         }
>>
>>         VarHandle handle = MemoryHandles.varHandle(long.class, 
>> ByteOrder.nativeOrder());
>>
>>         MappedMemorySegment segment = 
>> MemorySegment.mapFromPath(file.toPath(), 0, 128, 
>> FileChannel.MapMode.READ_WRITE);
>>
>>         MemorySegment sliceSegment = segment.asSlice(0, 8);
>>
>>         handle.set(sliceSegment.baseAddress(), 20);
>>
>>         MemorySegment unsafeSliceSegment = 
>> MemorySegment.ofNativeRestricted(
>>                 sliceSegment.baseAddress(),
>>                 CSupport.C_LONGLONG.byteSize(),
>>                 null,
>>                 null,
>>                 null
>>         );
>>
>>         segment.force();
>>         segment.unload();
>>         segment.close();
>>
>>         System.out.println("Big Segment Alive? " + 
>> segment.isAlive()); // prints false
>>         System.out.println("Sliced Segment Alive? " + 
>> sliceSegment.isAlive()); // prints false
>>         System.out.println("Unsafe Sliced Segment Alive? " + 
>> unsafeSliceSegment.isAlive()); // prints true
>>
>>         // Removing this line gives a JVM crash on access of 
>> unsafeSliceSegment
>>         segment = MemorySegment.mapFromPath(file.toPath(), 0, 128, 
>> FileChannel.MapMode.READ_WRITE);
>>
>> System.out.println(handle.get(unsafeSliceSegment.baseAddress()));
>>