Bug: MappedMemorySegmentImpl uses internal FileChannelImpl

[Maurizio Cimadamore]

On 01/07/2020 21:49, Ty Young wrote:
>
> On 6/29/20 1:04 PM, Maurizio Cimadamore wrote:
>>
>> On 29/06/2020 18:49, Ty Young wrote:
>>>
>>> On 6/29/20 11:32 AM, Maurizio Cimadamore wrote:
>>>> Note that to be able to map a file we need a file descriptor. The 
>>>> file system you provided has support for channels, etc. but note 
>>>> that the channels implementation do not support mapping using 
>>>> MappedByteBuffer:
>>>>
>>>> https://github.com/google/jimfs/blob/master/jimfs/src/main/java/com/google/common/jimfs/JimfsFileChannel.java#L592 
>>>>
>>>>
>>>> So I don't really see how we could make this work. At the end of 
>>>> the day, MemorySegment.mapFromPath has to end up calling `mmap` - 
>>>> and we need to be able to retrieve the parameters of the mmap call 
>>>> from somewhere.
>>>
>>>
>>> Would FMA having its own platform independant mmap bindings be a 
>>> possible solution? I guess I could just do it myself either way.
>>
>> I think I showed an example of how to create a mapped segment 
>> manually, by relying on FFI:
>>
>> https://gist.github.com/mcimadamore/128ee904157bb6c729a10596e69edffd
>>
>> If you happen to have a file descriptor for the thing you want to 
>> map, then an approach like this would be feasible
>
>
> To be clear, will FMA ever stop using internal details and what I'm 
> trying to achieve be possible using *just* JDK APIs? Surely FMA 
> shouldn't dictate this and lockout every custom FileSystem 
> implementation?

I think you just didn't understand what doesn't work with the custom 
file system implementation you were trying to use.

That file system implementation did *not* support memory mapping - even 
with byte buffers. I'm not sure what you expect should happen in that 
case, honestly. FMA can't outsmart the file system provider itself, or 
guess which file descriptor should be passed down to mmap.

Your tone isn't helping either - you make it sound like it's something 
that should _obviously_ work, while, to me it doesn't seem like you have 
fully grasped what it is that you are trying to do - other than the fact 
that you happen to get a Path from a custom file system and you want it 
to magically be memory mapped - somehow.

For that to happen, the mapped entity needs to be backed by a file 
descriptor (in Java, an opaque FileDescriptor instance). You are 
conflating multiple issues in the same email (not the first time 
either); the problem you see has literally _nothing_ to do with FMA - 
there's no concept of "memory mappable FileSystem" - which could give 
enough info to FMA (or byte buffer) to construct a memory mapped 
segment/buffer. There's just Paths; from a Path you can get a 
FileChannel - but again there's no constraint for a random FileChannel 
implementation to dole out file descriptors. The JDK ones do - and 
that's why you can map them. Most of the FS provider you find around 
don't, so I don't exactly know how the FMA, alone, is supposed to bridge 
that gap.

That said, the particular problem you are discussing (you are creating a 
memory mapped file out of a path obtained from an in-memory file system 
- so, in a way you are going in circles, by mapping into memory what was 
already in memory in the first place), is so removed from reality that I 
don't think it's worth discussing it further, certainly not in this way, 
sorry. I've offered many alternatives, from using realloc, to using 
custom mapped segments. With the FFI support you could even reserve 
(mmap, again) and commit (mprotect) your own memory and build an 
allocator which behaves exactly how you want (and will probably perform 
way better than using custom file system + mapped segments). Is that 
easy code to write? Probably not, but let's not pretend that what you 
are trying to achieve falls in the 80 part of the 80-20 use case mark 
[1], as you make it sound.

Maurizio

[1] - https://en.wikipedia.org/wiki/Pareto_principle



>
>
>>
>> Maurizio
>>
>>>
>>>
>>>>
>>>> Maurizio
>>>>
>>>>
>>>> On 29/06/2020 16:55, Ty Young wrote:
>>>>> Tried using Google's in-memory filesystem:
>>>>>
>>>>>
>>>>> https://github.com/google/jimfs
>>>>>
>>>>>
>>>>> for storing mmap files but ran into an issue because 
>>>>> MappedMemorySegmentImpl uses internal FileChannelImpl method:
>>>>>
>>>>>
>>>>> https://github.com/openjdk/panama-foreign/blob/acc11115db42cc0e17efdf361d58c65f4e922f39/src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/MappedMemorySegmentImpl.java#L105 
>>>>>
>>>>>
>>>>>
>>>>> which causes a class casting exception when using Google's 
>>>>> in-memory Java filesystem and probably other implementations as well.
>>>>>
>>>>>
>>>>>