[foreign] Panama EA - August 2019 edition

[Maurizio Cimadamore]

On 30/08/2019 21:22, Maurizio Cimadamore wrote:
>
> On 30/08/2019 21:12, Maurizio Cimadamore wrote:
>>
>> On 30/08/2019 20:07, Ty Young wrote:
>>>
>>> On 8/30/19 12:54 PM, Maurizio Cimadamore wrote:
>>>>
>>>> On 30/08/2019 18:41, Ty Young wrote:
>>>>>
>>>>> On 8/30/19 12:25 PM, Maurizio Cimadamore wrote:
>>>>>>
>>>>>> On 30/08/2019 17:59, Ty Young wrote:
>>>>>>>
>>>>>>> On 8/28/19 5:11 AM, Maurizio Cimadamore wrote:
>>>>>>>> I just want to clarify on this point below; the foreign memory 
>>>>>>>> access work does not, in any way, hinder the higher level 
>>>>>>>> functionalities provided by the foreign API/binder. We arrived 
>>>>>>>> at the foreign memory access API because we felt that something 
>>>>>>>> low level was missing - e.g. that the high level Pointer API 
>>>>>>>> was doing too much at once; and that users not interested in a 
>>>>>>>> high-level API, but still wanting to access off-heap data would 
>>>>>>>> not be served very well by the Pointer API alone.
>>>>>>>>
>>>>>>>> So, moving forward you can expect the bulk of the foreign API 
>>>>>>>> to be relatively stable (well, it's a prototype, so we might 
>>>>>>>> tweak things here and there); what will really change is how 
>>>>>>>> this API is _implemented_ - that is, moving forward the foreign 
>>>>>>>> API will be built _on top_ of the lower memory access and ABI 
>>>>>>>> layers. But high-level use cases using jextract need not to 
>>>>>>>> worry about this.
>>>>>>>>
>>>>>>>> I hope this clarifies better where we'd like to land.
>>>>>>>
>>>>>>>
>>>>>>> Yes, it does greatly. Thanks for clarifying.
>>>>>>>
>>>>>>>
>>>>>>> Speaking of the Pointer API, could a method be added to wrap a 
>>>>>>> pointer in another pointer for **char string pointers? AFAIK, 
>>>>>>> the only way to do that is:
>>>>>>>
>>>>>>>
>>>>>>> <LIB>.scope().allocate(<LIB>.scope().allocateCString("").type().pointer()); 
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ...which gets the Pointer<Pointer<Byte>> type that I need but 
>>>>>>> I'm not entirely sure if this is the correct way to go about 
>>>>>>> getting the type. Using the layout of a throwaway pointer layout 
>>>>>>> just seems wrong and wasteful.
>>>>>>
>>>>>> If you want to allocate a Pointer<Pointer<Byte>> you can do this:
>>>>>>
>>>>>>
>>>>>> <LIB>.scope().allocate(NativeTypes.UINT8.pointer());
>>>>>>
>>>>>
>>>>> Is the size the same as a C String with that?
>>>>
>>>> When you allocate a C string using allocateCString, you get back a 
>>>> Pointer<Byte>; this has an 'address' layout - that is, the size of 
>>>> this value is platform dependent, but on x64 platforms you can 
>>>> assume it's 64 bits.
>>>>
>>>> So, when you have a Pointer<Byte> and call type() on it, as in your 
>>>> previous code, you get the pointee layout type - that for 'Byte' 
>>>> (which is 8 bits). At which point you are calling pointer() on it, 
>>>> which is sending you back to a pointer layout type, with size 64 
>>>> (again assuming we're on x64). So, the outer allocate will allocate 
>>>> 64 bits.
>>>>
>>>> The code I suggested does the same - but without the throwaway 
>>>> allocation.
>>>>
>>>>
>>>> That said - I think what you really wanted to ask, is another 
>>>> question, one that has been raised before: assuming I have a 
>>>> Pointer<Byte> representing a C string, how do I get a pointer to that?
>>>>
>>>> Am I correct?
>>>
>>>
>>> Yes.
>>>
>>>
>>> The function calls for a **char(documented as a string) but there is 
>>> no obvious way to wrap the type returned by 
>>> allocateCString(Pointer<Byte>) into another 
>>> Pointer(Pointer<Pointer<Byte>>), hence the wasteful code above.
>>
>> So, let's say we have this:
>>
>> Pointer<Byte> c_str = scope.allocateCString("Hello");
>>
>> Now we have to create a pointer to that pointer. To do that:
>>
>> Pointer<Pointer<Byte>> p_c_str = 
>> scope.allocate(NativeTypes.UINT8.pointer());
>>
>> And then initialize the contents of the pointer to pointer:
>>
>> p_c_str.set(c_str);
>>
>> You can now pass p_c_str to your function.
>>
>> Is this what you wanted to do?
>
> And, I think here what bugs you is the need to allocate the outer 
> pointer - you want something akin to C's & operator.
>
> I made a proposal few months ago - which I've been able to find :-)
>
> https://mail.openjdk.java.net/pipermail/panama-dev/2019-February/004629.html 
>
>
> Didn't get much traction back then - but maybe we should revisit?

For the records, the proposal didn't go ahead as a result of an 
accidentally private discussion between me and Jorn. The conclusion can 
be summarized as follows:

For Pointer, Arrays and Callback, it is hard to add an addressOf method 
because you need to allocate (so you need to take a scope) - e.g. to do 
something akin to what I showed you in the other email.

Structs has already an addressof (called ptr), so, not much to gain 
there (other than the renaming)

Which basically left us with the feeling that there's not much that can 
be improved.

Maurizio


>
> Maurizio
>
>>
>> Maurizio
>>
>>>
>>>
>>>>
>>>> Maurizio
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Maurizio
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Maurizio
>>>>>>>>
>>>>>>>> On 19/08/2019 10:33, sundararajan.athijegannathan at oracle.com 
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> All that said, how close is Panama? Is this foreign memory 
>>>>>>>>>> API going to stay going forward or will the project take a 
>>>>>>>>>> major shift? I'd *really* like to start putting this to use 
>>>>>>>>>> and am willing to make adjustment where needed if minor 
>>>>>>>>>> changes are made, but if the entire foreign API is scrapped 
>>>>>>>>>> it isn't worth it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Panama "memory access" ("memaccess" panama-dev branch) API is 
>>>>>>>>> expected to become stable first and then other parts of 
>>>>>>>>> java.foreign later ("foreign" branch stuff).