[nicl] branch is now closed

[Maurizio Cimadamore]

Hi Samuel,
the implementation of the [foreign] branch reflects the document that 
has been circulating over the last few months:

http://cr.openjdk.java.net/~mcimadamore/panama/panama-binder-v3.html

I'm trying to keep that document up to date with developments in the 
repo, so that it matches the status quo.

As for the old implementation in the nicl branch, I'm afraid we have no 
single document that capture its state, other than the list of documents 
I've shared in the past.

But some of the things which have improved more markedly in the new 
branch are:

* array access - we now have a new Array<T> carrier to model native 
arrays; the binder turns arrays into instances of this class (and 
viceversa) - no deep copy required, and works on every type (in nicl, 
only a handful of types were implemented)

* function pointers - function pointers now work in all contexts, both 
as callback arguments to function AND as fields to structs (but we know 
we need to do more here to control the lifecycle of these things)

* much deeper macro support; in nicl only string and numeric constants 
were supported; we now allow arithmetic expression, parenthesis, etc.

* bitfields are now supported

* jextract generates correct amount of padding, so that alignment 
matches that seen by the C compiler at extraction time

* completely revamped layout language (as per document) backed up by a 
public layout API

* public API is fully documented (there's much to do here in order to 
improve both the API and the doc, but it's a start)

* many jextract bug fixes to iron out corner cases of the C language

* the binder has almost been rewritten from scratch; the new code 
generation model should allow for far better performances (as most 
operations are achieved by performing an invokeExact on a constant 
method handle).

These are the main differences between the two repo and what we have 
been working over the last few months.


As to what are we planning to do: we want to keep ironing out issues in 
jextract and in the binder - we are mostly focusing on 3 areas:

* lifecycle of function callbacks. The automatic conversion between Java 
functional interface and function pointers is cool, but seems to magic: 
it makes the translation non-raw (e.g. setting a function pointer on a 
struct field requires native memory allocation) - and it is not clear as 
to when things in memory can be collected (as these objects are 
synthesized on the fly, in most cases the user doesn't have an hard 
reference on them).

* Unresolved layout resolution scheme; as layout can point to other 
layouts, we need to define more precisely what that means, and what is 
the context in which this is possible. This is also related to the point 
below...

* Significantly redesign jextract to allow for better ease of use; while 
what we have is good for prototyping purposes, there are too many cases 
where, in order to get a library working, a lot of fiddling needs to be 
done, to exclude symbols, or just to have the right set of symbols 
included in the jextract artifacts. The model of supporting multiple 
jextract runs is also not clear - e.g. can we extract a library A, then 
extract a library B and pass the artifacts for A during the second run 
(e.g. similar to passing a classpath or modulepath to javac) ? We are 
having some discussion on these topics - and I'll share something more 
soon. The goal here is to create an experience that is as close as 
possible to that you get when interacting with standard C/C++ compiler 
toolchains.


Once we get some progress going on the tasks mentioned above, our plan 
is to start the process for releasing early access binaries (Linux/MacOs 
x64 only, for starters), so that people can play with the bits, without 
having to build the entire repo from scratch. I expect that this will 
generate some feedback which will trigger some further iterations on the 
API/toolchain.

Thanks
Maurizio






On 16/08/18 22:57, Samuel Audet wrote:
> Congratulations! Is there a changelog or a summary with a list of the 
> differences? It's hard to keep track of what has been implemented from 
> just the messages here. Documentation would be great, but I know 
> there's a lot to be done, so thanks for all the work!
>
> Samuel
>
> On 08/17/2018 12:46 AM, Maurizio Cimadamore wrote:
>> Dear citizens of the Isthmus,
>> as you might have noticed, over the last few months all the 
>> development activity regarding foreign data/function interface has 
>> moved from the 'nicl' branch to the 'foreign' branch. We have 
>> achieved feature parity between the two branches about two months 
>> ago; right now the set of functionalities available in the foreign 
>> branch greatly surpasses those provided by the legacy nicl branch, 
>> which is no longer worked on.
>>
>> For this reason, and also to minimize confusion and potential for 
>> errors, we have just closed the legacy 'nicl' branch. This doesn't 
>> mean the code has disappeared - you can in fact still update to 
>> 'nicl' (hg update nicl); but the 'nicl' branch will no longer be 
>> shown in the list of active branches (by the 'hg branches' command).
>>
>> Cheers
>> Maurizio
>>
>