[foreign] Panama EA - August 2019 edition

[Jorn Vernee]

On 30/08/2019 22:34, Maurizio Cimadamore wrote:
>
> On 30/08/2019 21:22, Maurizio Cimadamore wrote:
>>
>> On 30/08/2019 21:12, Maurizio Cimadamore wrote:
>>>
>>> On 30/08/2019 20:07, Ty Young wrote:
>>>>
>>>> On 8/30/19 12:54 PM, Maurizio Cimadamore wrote:
>>>>>
>>>>> On 30/08/2019 18:41, Ty Young wrote:
>>>>>>
>>>>>> On 8/30/19 12:25 PM, Maurizio Cimadamore wrote:
>>>>>>>
>>>>>>> On 30/08/2019 17:59, Ty Young wrote:
>>>>>>>>
>>>>>>>> On 8/28/19 5:11 AM, Maurizio Cimadamore wrote:
>>>>>>>>> I just want to clarify on this point below; the foreign memory 
>>>>>>>>> access work does not, in any way, hinder the higher level 
>>>>>>>>> functionalities provided by the foreign API/binder. We arrived 
>>>>>>>>> at the foreign memory access API because we felt that 
>>>>>>>>> something low level was missing - e.g. that the high level 
>>>>>>>>> Pointer API was doing too much at once; and that users not 
>>>>>>>>> interested in a high-level API, but still wanting to access 
>>>>>>>>> off-heap data would not be served very well by the Pointer API 
>>>>>>>>> alone.
>>>>>>>>>
>>>>>>>>> So, moving forward you can expect the bulk of the foreign API 
>>>>>>>>> to be relatively stable (well, it's a prototype, so we might 
>>>>>>>>> tweak things here and there); what will really change is how 
>>>>>>>>> this API is _implemented_ - that is, moving forward the 
>>>>>>>>> foreign API will be built _on top_ of the lower memory access 
>>>>>>>>> and ABI layers. But high-level use cases using jextract need 
>>>>>>>>> not to worry about this.
>>>>>>>>>
>>>>>>>>> I hope this clarifies better where we'd like to land.
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes, it does greatly. Thanks for clarifying.
>>>>>>>>
>>>>>>>>
>>>>>>>> Speaking of the Pointer API, could a method be added to wrap a 
>>>>>>>> pointer in another pointer for **char string pointers? AFAIK, 
>>>>>>>> the only way to do that is:
>>>>>>>>
>>>>>>>>
>>>>>>>> <LIB>.scope().allocate(<LIB>.scope().allocateCString("").type().pointer()); 
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ...which gets the Pointer<Pointer<Byte>> type that I need but 
>>>>>>>> I'm not entirely sure if this is the correct way to go about 
>>>>>>>> getting the type. Using the layout of a throwaway pointer 
>>>>>>>> layout just seems wrong and wasteful.
>>>>>>>
>>>>>>> If you want to allocate a Pointer<Pointer<Byte>> you can do this:
>>>>>>>
>>>>>>>
>>>>>>> <LIB>.scope().allocate(NativeTypes.UINT8.pointer());
>>>>>>>
>>>>>>
>>>>>> Is the size the same as a C String with that?
>>>>>
>>>>> When you allocate a C string using allocateCString, you get back a 
>>>>> Pointer<Byte>; this has an 'address' layout - that is, the size of 
>>>>> this value is platform dependent, but on x64 platforms you can 
>>>>> assume it's 64 bits.
>>>>>
>>>>> So, when you have a Pointer<Byte> and call type() on it, as in 
>>>>> your previous code, you get the pointee layout type - that for 
>>>>> 'Byte' (which is 8 bits). At which point you are calling pointer() 
>>>>> on it, which is sending you back to a pointer layout type, with 
>>>>> size 64 (again assuming we're on x64). So, the outer allocate will 
>>>>> allocate 64 bits.
>>>>>
>>>>> The code I suggested does the same - but without the throwaway 
>>>>> allocation.
>>>>>
>>>>>
>>>>> That said - I think what you really wanted to ask, is another 
>>>>> question, one that has been raised before: assuming I have a 
>>>>> Pointer<Byte> representing a C string, how do I get a pointer to 
>>>>> that?
>>>>>
>>>>> Am I correct?
>>>>
>>>>
>>>> Yes.
>>>>
>>>>
>>>> The function calls for a **char(documented as a string) but there 
>>>> is no obvious way to wrap the type returned by 
>>>> allocateCString(Pointer<Byte>) into another 
>>>> Pointer(Pointer<Pointer<Byte>>), hence the wasteful code above.
>>>
>>> So, let's say we have this:
>>>
>>> Pointer<Byte> c_str = scope.allocateCString("Hello");
>>>
>>> Now we have to create a pointer to that pointer. To do that:
>>>
>>> Pointer<Pointer<Byte>> p_c_str = 
>>> scope.allocate(NativeTypes.UINT8.pointer());
>>>
>>> And then initialize the contents of the pointer to pointer:
>>>
>>> p_c_str.set(c_str);
>>>
>>> You can now pass p_c_str to your function.
>>>
>>> Is this what you wanted to do?
>>
>> And, I think here what bugs you is the need to allocate the outer 
>> pointer - you want something akin to C's & operator.
>>
>> I made a proposal few months ago - which I've been able to find :-)
>>
>> https://mail.openjdk.java.net/pipermail/panama-dev/2019-February/004629.html 
>>
>>
>> Didn't get much traction back then - but maybe we should revisit?
>
> For the records, the proposal didn't go ahead as a result of an 
> accidentally private discussion between me and Jorn. The conclusion 
> can be summarized as follows:
>
> For Pointer, Arrays and Callback, it is hard to add an addressOf 
> method because you need to allocate (so you need to take a scope) - 
> e.g. to do something akin to what I showed you in the other email.
>
> Structs has already an addressof (called ptr), so, not much to gain 
> there (other than the renaming)
>
> Which basically left us with the feeling that there's not much that 
> can be improved.

FWIW, Array also has a ptr() method nowadays that returns a 
Pointer<Array<X>>.

Struct and Array are pretty easy to add such a method to, since they are 
implemented as a pointer to a block of memory that holds their value 
already. Pointer on the other hand stores it's value directly in a Java 
`long`. It is not really possible to return a 'stable' Pointer to that 
`long`; e.g. the Pointer object that holds the `long` might reside on 
the GC heap, and be moved from time to time, invalidating any Pointer 
that points to it. Even objects located on the stack are not guaranteed 
to be in the same place all the time (think of Loom's continuations).

So, you'd need to put this value in a stable place, e.g. off-heap. But, 
this also means that any pointer to the 'stable' value is not actually a 
Pointer to the Pointer you already had, but rather to a copy of it. i.e. 
Updating the value through this new Pointer will not affect the value of 
the original.

Maybe there is another way to get around these problems, but making 
things explicit in the written code also has it's merits, since you can 
_see_ what's going on, and don't have to memorize several corner cases 
in which things happen differently than you would expect.

Jorn

>
> Maurizio
>
>
>>
>> Maurizio
>>
>>>
>>> Maurizio
>>>
>>>>
>>>>
>>>>>
>>>>> Maurizio
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Maurizio
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Maurizio
>>>>>>>>>
>>>>>>>>> On 19/08/2019 10:33, sundararajan.athijegannathan at oracle.com 
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> All that said, how close is Panama? Is this foreign memory 
>>>>>>>>>>> API going to stay going forward or will the project take a 
>>>>>>>>>>> major shift? I'd *really* like to start putting this to use 
>>>>>>>>>>> and am willing to make adjustment where needed if minor 
>>>>>>>>>>> changes are made, but if the entire foreign API is scrapped 
>>>>>>>>>>> it isn't worth it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Panama "memory access" ("memaccess" panama-dev branch) API is 
>>>>>>>>>> expected to become stable first and then other parts of 
>>>>>>>>>> java.foreign later ("foreign" branch stuff).