Ability to extend a MemorySegment

[Maurizio Cimadamore]

On 21/01/2020 04:54, Ty Young wrote:
>
> On 1/20/20 7:26 PM, Maurizio Cimadamore wrote:
>>
>> On 21/01/2020 01:07, Ty Young wrote:
>>>
>>> On 1/20/20 6:32 PM, Maurizio Cimadamore wrote:
>>>> A memory segment has a fixed size - its boundaries are set on 
>>>> construction (pretty much like in a ByteBuffer). Think of a 
>>>> MemorySegment as the result of a 'malloc' - if you want something 
>>>> fancier than that, it's likely you're trying to use a memory 
>>>> segment in a way it's not intended to.
>>>
>>>
>>> How exactly is expanding the length of a smaller MemorySegment into 
>>> a larger MemorySegment not an intended use? Even beyond arrays, 
>>> there are plenty of cases where one might, for example, need to swap 
>>> an Integer MemorySegment into a Long MemorySegment or something 
>>> larger for a specific platform. Surely using realloc would be 
>>> faster/better?
>>
>> If you meant "why isn't realloc there?" I think I've replied - that 
>> can indeed be added. Is it something which should have held back the 
>> API for getting released as incubating in JDK 14? Probably not.
>
>
> I suppose it isn't an immediate concern but rather falls into the 
> giant TO-DO list like implementing/adding/exposing all the missing 
> infrastructure API. If Memory Access can be put into "incubator" 
> status despite no way to determine the underlying SystemABI then that 
> seems like a somewhat low threshold, IMO.

You seem to keep forgetting that what has been put into incubating state 
is the memory access API, period. Not the ABI part. The memory access 
part is useful on its own, to replace certain cases where the ByteBuffer 
API might not be suitable (e.g. working with persistent memory files 
bigger than 2G).

The ABI part will be incubating when it's ready - we've started making 
progress on fixing few issues just now.

>
>
>>
>>
>>>
>>>
>>>>
>>>> So, when you use a layout to do MemorySegment::allocateNative, the 
>>>> layout is used to determine the size (and alignment) of the chunk 
>>>> of memory to be allocated - so the size has to be known at segment 
>>>> creation.
>>>>
>>>> How would an array list be implemented in C? Or, how is it 
>>>> implemented even in Java? You will see that the 'creating new array 
>>>> and copying contents' is really how things are done at the low 
>>>> level. So, if you want a variable-sized data structure based on 
>>>> MemorySegment, yes, you will have to take into account the fact 
>>>> that the MemorySegment will have to be thrown away and re-created 
>>>> with a new size (**)
>>>>
>>>> (*) Actually there is another possibility when working with 
>>>> malloc/free which is "realloc" - it is theoretically possible to 
>>>> provide a re-alloc API point to memory segment which would:
>>>>
>>>> - kill the current segment
>>>> - realloc the underlying native memory to new size
>>>> - create a new segment with new size
>>>>
>>>> But we will have to evaluate this possibility in conjunction with 
>>>> the fact that, one day, memory segment might not exactly map to 
>>>> malloc/free and instead use a different allocator (which might or 
>>>> might not have ability to "reallocate"). Something to keep in mind 
>>>> though.
>>>
>>>
>>> Isn't it the expectation and/or assumption that Panama will use 
>>> standard C allocators and not some other arbitrary allocator anyway? 
>>> I understand it *technically* falls into the implementation detail 
>>> bucket but malloc/free/realloc are standard C library functions...
>>
>> There is a desire to have some allocator whose performances are more 
>> similar to stack allocation than to native heap allocation, at least 
>> in cases where memory is allocated for data to be passed to a 
>> function. Some discussion on this topic can be found here:
>>
>> https://mail.openjdk.java.net/pipermail/core-libs-dev/2019-November/063695.html 
>>
>>
>> and also here:
>>
>> https://mail.openjdk.java.net/pipermail/panama-dev/2019-November/006748.html 
>>
>>
>> Finally, note that, regardless of which allocator we end up using by 
>> default, malloc/free/realloc are just... native calls, and they can 
>> be wrapped as method handles - and you can even create segments with 
>> the right spatial bounds to back them up so that out-of-bound access 
>> is disallowed. (**)
>>
>> (**) This reminds me that our ForeignUnsafe::uncheckedSegment should 
>> take an extra optional parameter - the callback function to be called 
>> when the segment is closed.
>
>
> Yes, but having all of that as part of the proper API would be more 
> pleasant.

Also remind that a MemorySegment can model both an offheap, heap, memory 
mapped, byte bufer segment. Not all of these might have an associated 
meaning for 'reallocate'. It's not a blocker, but it's definitively 
something that has its own relevance at the API level.

Maurizio

>
>
>>
>> Maurizio
>>
>>>
>>>
>>>>
>>>> Maurizio
>>>>
>>>>
>>>>
>>>>
>>>> On 20/01/2020 21:53, Ty Young wrote:
>>>>> Hi,
>>>>>
>>>>>
>>>>> Would it be possible to add the ability to extend a memory 
>>>>> segment(in other words, increase its limit)? Right now it's not 
>>>>> possible to implement dynamic arrays(read: ArrayList like arrays) 
>>>>> as it's not possible to calculate the length of a SequenceLayout 
>>>>> that has no specified length. You can work around this by creating 
>>>>> a new array and copying the contents of the old array to it but 
>>>>> that seems inefficient...
>>>>>