Ability to extend a MemorySegment

[Ty Young]

On 1/20/20 7:26 PM, Maurizio Cimadamore wrote:
>
> On 21/01/2020 01:07, Ty Young wrote:
>>
>> On 1/20/20 6:32 PM, Maurizio Cimadamore wrote:
>>> A memory segment has a fixed size - its boundaries are set on 
>>> construction (pretty much like in a ByteBuffer). Think of a 
>>> MemorySegment as the result of a 'malloc' - if you want something 
>>> fancier than that, it's likely you're trying to use a memory segment 
>>> in a way it's not intended to.
>>
>>
>> How exactly is expanding the length of a smaller MemorySegment into a 
>> larger MemorySegment not an intended use? Even beyond arrays, there 
>> are plenty of cases where one might, for example, need to swap an 
>> Integer MemorySegment into a Long MemorySegment or something larger 
>> for a specific platform. Surely using realloc would be faster/better?
>
> If you meant "why isn't realloc there?" I think I've replied - that 
> can indeed be added. Is it something which should have held back the 
> API for getting released as incubating in JDK 14? Probably not.


I suppose it isn't an immediate concern but rather falls into the giant 
TO-DO list like implementing/adding/exposing all the missing 
infrastructure API. If Memory Access can be put into "incubator" status 
despite no way to determine the underlying SystemABI then that seems 
like a somewhat low threshold, IMO.


>
>
>>
>>
>>>
>>> So, when you use a layout to do MemorySegment::allocateNative, the 
>>> layout is used to determine the size (and alignment) of the chunk of 
>>> memory to be allocated - so the size has to be known at segment 
>>> creation.
>>>
>>> How would an array list be implemented in C? Or, how is it 
>>> implemented even in Java? You will see that the 'creating new array 
>>> and copying contents' is really how things are done at the low 
>>> level. So, if you want a variable-sized data structure based on 
>>> MemorySegment, yes, you will have to take into account the fact that 
>>> the MemorySegment will have to be thrown away and re-created with a 
>>> new size (**)
>>>
>>> (*) Actually there is another possibility when working with 
>>> malloc/free which is "realloc" - it is theoretically possible to 
>>> provide a re-alloc API point to memory segment which would:
>>>
>>> - kill the current segment
>>> - realloc the underlying native memory to new size
>>> - create a new segment with new size
>>>
>>> But we will have to evaluate this possibility in conjunction with 
>>> the fact that, one day, memory segment might not exactly map to 
>>> malloc/free and instead use a different allocator (which might or 
>>> might not have ability to "reallocate"). Something to keep in mind 
>>> though.
>>
>>
>> Isn't it the expectation and/or assumption that Panama will use 
>> standard C allocators and not some other arbitrary allocator anyway? 
>> I understand it *technically* falls into the implementation detail 
>> bucket but malloc/free/realloc are standard C library functions...
>
> There is a desire to have some allocator whose performances are more 
> similar to stack allocation than to native heap allocation, at least 
> in cases where memory is allocated for data to be passed to a 
> function. Some discussion on this topic can be found here:
>
> https://mail.openjdk.java.net/pipermail/core-libs-dev/2019-November/063695.html 
>
>
> and also here:
>
> https://mail.openjdk.java.net/pipermail/panama-dev/2019-November/006748.html 
>
>
> Finally, note that, regardless of which allocator we end up using by 
> default, malloc/free/realloc are just... native calls, and they can be 
> wrapped as method handles - and you can even create segments with the 
> right spatial bounds to back them up so that out-of-bound access is 
> disallowed. (**)
>
> (**) This reminds me that our ForeignUnsafe::uncheckedSegment should 
> take an extra optional parameter - the callback function to be called 
> when the segment is closed.


Yes, but having all of that as part of the proper API would be more 
pleasant.


>
> Maurizio
>
>>
>>
>>>
>>> Maurizio
>>>
>>>
>>>
>>>
>>> On 20/01/2020 21:53, Ty Young wrote:
>>>> Hi,
>>>>
>>>>
>>>> Would it be possible to add the ability to extend a memory 
>>>> segment(in other words, increase its limit)? Right now it's not 
>>>> possible to implement dynamic arrays(read: ArrayList like arrays) 
>>>> as it's not possible to calculate the length of a SequenceLayout 
>>>> that has no specified length. You can work around this by creating 
>>>> a new array and copying the contents of the old array to it but 
>>>> that seems inefficient...
>>>>