notes on binding C++

Thu Feb 8 12:27:26 UTC 2018

Hi Samuel,
some points inline below:

On 08/02/18 03:44, Samuel Audet wrote:
> On 02/06/2018 08:19 PM, Maurizio Cimadamore wrote:
>> On 06/02/18 07:52, Samuel Audet wrote:
>>> That sounds great and all, but it has been almost 4 years now, and 
>>> the only feedback I am getting is that it does not yet support 
>>> function-like macros, inline functions, C++ templates, etc. Ok, 
>>> then, when? Why is a "limited amount of wrappers" taking so long?
>> Hi Samuel,
>> I think I've already responded to similar questions many times now, 
>> and I feel we're just going in circles; I sense your frustration, but 
>> unfortunately I do not have anything more to add to this discussion. 
>> I've told you what the focus is short/medium term (C interop), and 
>> what is the _rough_ plan after that. If that is not sufficient, or if 
>> you feel that we just got all the priorities wrong, or if you think 
>> we're just too slow, I'm sorry, but I don't think those kinds of 
>> concerns are of the kind that can be positively addressed over an 
>> email exchange.
>
> I am sorry, I think that came out the wrong way. What I was trying to 
> say is that supporting C++ and making it as super efficient as you 
> want it to be is a *very hard problem*, and you will require 
> assistance from a lot of people--from outside Oracle--to pull this 
> off. I am trying to make you realize that, and yes I am frustrated 
> that I am unable to communicate this, even after almost 4 years...

Well, I've only being on the receiving side of such communications for a 
couple of weeks of so :-)
While I fully understand that efficient C++ integration is an hard 
problem, I'm also (very) confident in the collective brain of people 
involved in this project (not just from Oracle!).

>
> Another constructive comment I would like to make is, would it be 
> possible to release all the relevant documents about jextract and 
> Panama so that people outside of Oracle could participate and make 
> this a community effort? If you need more data to convince managers 
> that this is required in order for this project to succeed, I can help 
> with that! However, as I mentioned previously, I feel that my help (or 
> the help from any outsiders for that matter) is not welcome here. It 
> would be great if this state of affairs could be improved! (This might 
> be for you, Bernard?)
The current state of affair in Panama/nicl land is not great and we're 
aware of that; while several documents have been produced and made 
available to the public from the start [1, 2, 3, 4, 5, 6, 7, 8], I feel 
that there's a disconnect between the high level view that these 
documents are trying to portrait (with the exception maybe of [3]), and 
what's currently implemented. Also, for many reasons (mainly pragmatic 
compromises to get to a proof of concept faster), the implementation 
took different choices from those that were described in that document. 
Some of these choices were unavoidable - and they are the natural part 
of turning a design into a product; other choices less so (especially 
those around layout description).

Let me also add that I'm fully aware that the code is not documented 
much - and that it is very hard to grasp what the API is, and the set of 
assumptions on which the current binder works - I've made an attempt few 
weeks ago [11], which is by no means meant to be exhaustive. But I'm 
acutely aware of all these issues - partly because, having recently 
joined this project, I had to go through the effort of understanding it 
- and it has not been exactly easy to do that.

It is precisely because of these issues that we're now trying to 
kickstart few discussions; you might have seen some emails published 
recently [9,
10] re. analysis of layout description in real world FFI and message 
protocol; I plan to post an email soon which will outline a set of 
requirements for layout descriptions which will form a contract between 
the Java code and the binder - and we plan to do a better job in 
documenting the code, as well as capturing the state of the 
implementation in documents in a more accurate way, so that it will be 
easier for other people to join in and play with what we're building.

That said, it would be unrealistic to promise that it will all change 
overnight - and there will be hiccups here and there as we try to 'steer 
the boat'; but I think, I've seen a lot of improvements in the last few 
months - maybe not all of them were noticeable or perceived as important 
from the outside - but the very fact that we're having this discussion 
after a relatively long 'quiet period', to me is an indication that 
we're moving in the right direction.

Regarding feedback, we definitively welcome all of it - no exceptions. 
That said, as I said once, we live in a world of limited resources, 
which means that we have to prioritize our work (and feedback too) if we 
want to succeed. The world of native interop is big and complex (as you 
correctly keep reminding us), and it would be unrealistic to assume that 
we will get there in one step (and we don't assume that, rest assured!). 
Which is why, for now, our main focus is C interop - to give a bit of 
relief to many Java developers who would like to use very good system 
libraries but are unable to do so because of either ease of use or 
performance concerns. As much as we'd like to dive deep into the problem 
of C++ interop, we're also aware that we'd be doing so at the expenses 
of our short/medium term goals - after all, there are only so many 
battles we can fight at the same time.

So, maybe it could be worth checking back in a few months, to see if 
things have progressed further - and if the project is more in a state 
where you feel like you could start contributing with more ease. That 
said, I think the points you raise about C++ interop are valid longer 
term points, and they should be discussed somewhere - but maybe there's 
a better venue for that discussion and that is the "panama-spec-experts" 
mailing list, which typically has a wider audience (this mailing list is 
primarily for discussing implementation aspects).

Cheers
Maurizio

[1] - http://cr.openjdk.java.net/~jrose/panama/minimal-ldl.html
[2] - http://cr.openjdk.java.net/~jrose/panama/isthmus-in-the-vm-2014.html
[3] - http://cr.openjdk.java.net/~jrose/panama/panama-status-2015-1216.pdf
[4] - http://cr.openjdk.java.net/~jrose/panama/native-call-primitive.html
[5] - http://cr.openjdk.java.net/~jrose/panama/cppapi.cpp.txt
[6] - 
http://cr.openjdk.java.net/~jrose/panama/Metadata-Flow-Possibilities.pdf
[7] - http://cr.openjdk.java.net/~jrose/panama/Metadata-Flow-User-View.pdf
[8] - http://cr.openjdk.java.net/~jrose/panama/using-interfaces.html
[9] - 
http://mail.openjdk.java.net/pipermail/panama-dev/2018-January/000915.html
[10] - 
http://mail.openjdk.java.net/pipermail/panama-dev/2018-February/000940.html
[11] - 
http://mail.openjdk.java.net/pipermail/panama-dev/2018-January/000919.html
>
> Samuel