Ability to extend a MemorySegment

[Ty Young]

On 1/21/20 5:22 AM, Michael Zucchi wrote:
> On 21/1/20 11:37 am, Ty Young wrote:
>>> A memory segment has a fixed size - its boundaries are set on 
>>> construction (pretty much like in a ByteBuffer). Think of a 
>>> MemorySegment as the result of a 'malloc' - if you want something 
>>> fancier than that, it's likely you're trying to use a memory segment 
>>> in a way it's not intended to.
>>
>>
>> How exactly is expanding the length of a smaller MemorySegment into a 
>> larger MemorySegment not an intended use? Even beyond arrays, there 
>> are plenty of cases where one might, for example, need to swap an 
>> Integer MemorySegment into a Long MemorySegment or something larger 
>> for a specific platform. Surely using realloc would be faster/better? 
>
> There's a much better solution to that case, just allocate the larger 
> of the possible items - no need to even check what the size is when 
> you use it.  This is exactly the approach one would use in C - it's 
> effectively a union.


That's an option, I suppose. Using a Union requires that you keep track 
of the valid value though which would require a higher level abstraction.


I've noticed some weirdness with Unions in Memory Access as well - 
creating a slice that is the exact size of the Union's 
MemoryAddress(long for example) and applying a value results in the 
Union's base MemoryAddress being made invalid(despite being the same 
size). Maybe that's intentional.


>   Seems an odd example though, it's not like you can migrate a running 
> application to a different platform and so you have to know all these 
> sizes before you get to that point.


NVML (what I'm using Panama for, partially anyway) is cross-platform. 
I'd rather write code once and never have tweak anything for Windows.


>
> Java's ArrayList never grows the array since you can't grow Java 
> arrays and that doesn't seem to have hurt it much for most uses. In 
> cases where it is a problem you just use a more appropriate data 
> structure.


True, but the whole point of doing off-heap memory is because it's 
faster than JVM managed memory.


>
> realloc isn't magic either.  It isn't guaranteed to do anything more 
> than alloc+copy+free.  The one in glibc tries not to do that but it 
> relies on there being a following free block that is big enough, and 
> once you get to larger allocations (128K by default in the source) it 
> switches to mmap blocks and relies on operating system support and 
> available holes in the virtual memory map.


If all the limit increasing API does is alloc+copy+free on most other 
platforms/allocators then so be it. If the allocator can be changed on a 
whim in the future then I don't see why It can't be implementation specific.


Maybe the allocator implementation could be swapped via a command line 
argument or something?


> for all that you can never actually use these array-based segments 
> with "foreign" functions anyway.


I'm able to pass the address of my Array<Byte> abstraction to NVML's 
systemGetDriverVersion and it fills it with "440"(my driver version)... 
sometimes. I had issues when I wrapped it in my Pointer<String> 
abstraction(which uses Array<Byte> under the hood) but that's probably 
just my own technical failure.