[foreign-abi] open issues

[Maurizio Cimadamore]

Below is a list of todo items for the system abi work that I've been 
collecting as I was going through the initial ABI work. I grouped the 
todo items in two buckets: the first has more to do with broad issues 
which can affect design and implementation, whereas the latter has more 
to do with low-level implementation details, and are unlikely to affect 
the overall design of the API.

High-level open issues:

* Layout vs. FunctionDescriptor should we unify?

* How to drive ABI classification: layout annotations, carriers or...

* How to handle varargs? Modelling them with Java varargs work for 
downcalls, not for upcalls (and even for downcalls, they work so so, see 
Win64). Do we need a valist-like type?

* Unify backends - we currently support a mixture of Universal/Direct 
upcall/downcall backend which vary depending on platforms. Link2Native 
should replace everything here

* Where to put the SystemABI API? It is an unsafe API (since abusing it 
can crash the VM) - we don't have a great place for APIs like this in 
the JDK. Classic answer has been to move things into sun.misc.unsafe - 
something better is needed here

* Should we support multiple calling conventions? What about 'custom' 
calling conventions?

Low-level open issues:

* Creating native memory addresses from scratch - what to do with 
segment bounds?

* Upcall segments - the size of the segment is currently just 8 bytes - 
should we return correct size (and mark the segment as readonly) ?

* Library lookup - MemoryAddress results should be read only AND pinned

* Are we satisfied with the way in which structs are returned by value - 
e.g. asking clients to be in charge of cleanup by closing the struct 
segment?

* We should deal with situations where the 'wrong' address/segment is 
being passed (e.g. one that is on-heap) in a graceful way


We have already started some discussions to address some of the high 
level bits (e.g. the role of layouts in driving ABI classification, see 
[1]), we have documents for some (e.g. vararags, see [2]), and ongoing 
work for others (e.g. linkToNative). I expect we'll move through this 
list (and maybe add more elements to it) over the next few weeks as we 
iterate over the foreign-abi branch. So, I think we're collectively 
working on the right items. I'd like to slightly prioritize some of the 
items being discussed in [1] (e.g. role of layouts etc.) as that has 
repercussions on the memory access API, which we should try to finalize.

Cheers
Maurizio

[1] - 
https://mail.openjdk.java.net/pipermail/panama-dev/2019-July/005975.html
[2] - 
https://mail.openjdk.java.net/pipermail/panama-dev/2019-April/005212.html
[3] - http://hg.openjdk.java.net/panama/dev/shortlog/635654084654