taming resource scopes

[Ty Young]

On 6/2/21 5:43 AM, Maurizio Cimadamore wrote:
>
> On 02/06/2021 11:33, Ty Young wrote:
>> MemorySegments/scopes don't prevent freed memory from being accessed 
>> If you enable restricted methods and invoke the CLinker.freeMemory 
>> method. This seems like a bug with the method itself since it's 
>> officially part of Panama's API, even if part of the "unsafe" part. 
>> Custom made free bindings may not automatically call close, of 
>> course, but CLinker.freeMemory maybe should?
> That's the meaning of restricted - if you call these methods, you 
> abandon safety guarantees. In the case of CLinker::freeMemory, the 
> reason this method (and his companion CLinker::allocateMemory) exist, 
> is to provide a plain wrapper around malloc/free. Sometimes native 
> code gives an address to you and expects you to free it. You don't 
> have a segment, nor a scope. You just have an address. The only wa


Then just returning a primitive address would probably be better then, 
right? Maybe an override could be added that accepts a primitive long 
and does what freeMemory does now while modifying the existing method 
that accepts a MemoryAddress to call close maybe?


free knows nothing about MemoryAddress or scopes yet the method takes in 
a MemoryAddress and doesn't invalidate the scope by calling close, all 
of which are Panama related and, again, have nothing to do with free. It 
just feels off since MemoryAddress isn't *just* a wrapper around a long 
anymore but has a scope tied to it.


I can do this myself with custom bindings, I guess.


>>
>>
>> On the topic of free memory, what's the deal with calling free from 
>> different bindings/methods? If you do:
>>
>>
>> MemorySegment segment = MemorySegment.allocateNative(4, 
>> ResourceScope.newSharedScope());
>>
>> CLinker.freeMemory(segment.address());
>> CLinker.freeMemory(segment.address());
>>
>>
>> or, with hand made bindings:
>>
>>
>> MemorySegment segment = MemorySegment.allocateNative(4, 
>> ResourceScope.newSharedScope());
>>
>> clib.free(segment.address());
>> clib.free(segment.address());
>>
>>
>> On Linux you get:
>>
>>
>> free(): double free detected in tcache 2
>>
>>
>> as expected but mixing free bindings:
>>
>>
>> MemorySegment segment = MemorySegment.allocateNative(4, 
>> ResourceScope.newSharedScope());
>>
>> clib.free(segment.address());
>> CLinker.freeMemory(segment.address());
>>
>>
>> gives nothing nor does using the same bindings in different 
>> methods.Is this expected?
>
> First, you should never mix safe segments with unsafe, restricted 
> code. CLinker::freeMemory is not something you should look at if you 
> are after writing safe code. I explained above when these methods are 
> meant to be used.
>
> The message you see come from the guts of `free`. As long as your 
> clib::free indeed point to the same free function, then the difference 
> is probably explained in terms of a failure in the heuristics which 
> detects double free.
>
> If your clib::free links a _different_ free method, then this is more 
> expected, as different allocators typically do not cooperate with each 
> other. But since you are seeing same problem with calling same free 
> binding on different methods, I'd put that down to a failure in 
> heuristics.
>
> But the point remains: CLinker::free offers (as you can see) no safety 
> guarantees - which is why it's a restricted method and why you have to 
> pass a flag on the command line to use it.


I'm more than fully aware about the safety guareantees. I just wasn't 
sure if this was an issue with Panama.


>
> Maurizio
>
>>
>>
>> On 6/2/21 3:16 AM, Chris Vest wrote:
>>> You can build an Arc (Atomic Reference Counted) like thing as a 
>>> library,
>>> but without deep language integration it will be easy for the contained
>>> reference to escape the Arc.
>>> Thankfully MemorySegments will still prevent you from accessing freed
>>> memory.
>>> An alternative to building a generic Arc is to build specialised 
>>> atomically
>>> reference counting wrappers for each thing you wish to have reference
>>> counted, and then delegate calls instead of exposing the inner 
>>> reference.
>>> You still run into problems that, without language integration, object
>>> references can be shared without any enforcement of counter 
>>> adjustments.
>>> This makes usage awkward and introduces a new class of bugs that can 
>>> occur
>>> in programs.
>>> You might also run into an issue that your object can now be in one of
>>> three states: owned, shared, and closed/freed/released – the 
>>> distinction
>>> between owned and shared is the new thing.
>>> Objects that are in a shared state might not support all of the 
>>> operations
>>> of an owned object.
>>> Rust tracks this information in the type system as part of the 
>>> language.
>>> I built a prototype buffer implementation where I tracked this 
>>> information
>>> at runtime, but discarded the idea because it is, again, awkward and 
>>> error
>>> prone.
>>> There were many places where the code had to check and branch on the
>>> owned/shared state, and failure to do so would cause exceptions at 
>>> runtime.
>>> So my point is that Rust and Swift have good reference counting stories
>>> because of specific language features, and without similar language
>>> features I don't think having a generic or general Arc class in Java 
>>> will
>>> be very helpful.
>>>
>>> Cheers,
>>> Chris