Ability to extend a MemorySegment

[Maurizio Cimadamore]

On 21/01/2020 01:07, Ty Young wrote:
>
> On 1/20/20 6:32 PM, Maurizio Cimadamore wrote:
>> A memory segment has a fixed size - its boundaries are set on 
>> construction (pretty much like in a ByteBuffer). Think of a 
>> MemorySegment as the result of a 'malloc' - if you want something 
>> fancier than that, it's likely you're trying to use a memory segment 
>> in a way it's not intended to.
>
>
> How exactly is expanding the length of a smaller MemorySegment into a 
> larger MemorySegment not an intended use? Even beyond arrays, there 
> are plenty of cases where one might, for example, need to swap an 
> Integer MemorySegment into a Long MemorySegment or something larger 
> for a specific platform. Surely using realloc would be faster/better?

If you meant "why isn't realloc there?" I think I've replied - that can 
indeed be added. Is it something which should have held back the API for 
getting released as incubating in JDK 14? Probably not.


>
>
>>
>> So, when you use a layout to do MemorySegment::allocateNative, the 
>> layout is used to determine the size (and alignment) of the chunk of 
>> memory to be allocated - so the size has to be known at segment 
>> creation.
>>
>> How would an array list be implemented in C? Or, how is it 
>> implemented even in Java? You will see that the 'creating new array 
>> and copying contents' is really how things are done at the low level. 
>> So, if you want a variable-sized data structure based on 
>> MemorySegment, yes, you will have to take into account the fact that 
>> the MemorySegment will have to be thrown away and re-created with a 
>> new size (**)
>>
>> (*) Actually there is another possibility when working with 
>> malloc/free which is "realloc" - it is theoretically possible to 
>> provide a re-alloc API point to memory segment which would:
>>
>> - kill the current segment
>> - realloc the underlying native memory to new size
>> - create a new segment with new size
>>
>> But we will have to evaluate this possibility in conjunction with the 
>> fact that, one day, memory segment might not exactly map to 
>> malloc/free and instead use a different allocator (which might or 
>> might not have ability to "reallocate"). Something to keep in mind 
>> though.
>
>
> Isn't it the expectation and/or assumption that Panama will use 
> standard C allocators and not some other arbitrary allocator anyway? I 
> understand it *technically* falls into the implementation detail 
> bucket but malloc/free/realloc are standard C library functions...

There is a desire to have some allocator whose performances are more 
similar to stack allocation than to native heap allocation, at least in 
cases where memory is allocated for data to be passed to a function. 
Some discussion on this topic can be found here:

https://mail.openjdk.java.net/pipermail/core-libs-dev/2019-November/063695.html

and also here:

https://mail.openjdk.java.net/pipermail/panama-dev/2019-November/006748.html

Finally, note that, regardless of which allocator we end up using by 
default, malloc/free/realloc are just... native calls, and they can be 
wrapped as method handles - and you can even create segments with the 
right spatial bounds to back them up so that out-of-bound access is 
disallowed. (**)

(**) This reminds me that our ForeignUnsafe::uncheckedSegment should 
take an extra optional parameter - the callback function to be called 
when the segment is closed.

Maurizio

>
>
>>
>> Maurizio
>>
>>
>>
>>
>> On 20/01/2020 21:53, Ty Young wrote:
>>> Hi,
>>>
>>>
>>> Would it be possible to add the ability to extend a memory 
>>> segment(in other words, increase its limit)? Right now it's not 
>>> possible to implement dynamic arrays(read: ArrayList like arrays) as 
>>> it's not possible to calculate the length of a SequenceLayout that 
>>> has no specified length. You can work around this by creating a new 
>>> array and copying the contents of the old array to it but that seems 
>>> inefficient...
>>>