[nicl] branch is now closed

Fri Aug 17 23:16:05 UTC 2018

Thanks for the summary! That's what I was looking for.

About the Array type, it's still not clear to me why we can't just 
extend the Pointer with a "length"? It's not like pointers in C don't 
typically point to memory regions with length > 0: It's just often 
unknown. I've modeled the Pointer type in JavaCPP after Buffer (with 
position, limit, and capacity) and I haven't had any issues at all in 
years. None whatsoever. What is the benefit you perceive in mapping 
artificially 1 C concept into 2 Java concepts? I was not able to find 
the rationale from your document... If the idea is to provide safe 
get()/set() methods, why not simply use a layout? What does the Array 
type do that a layout could not do? That's basically what I do with the 
indexer package and it works fine:
http://bytedeco.org/news/2014/12/23/third-release/
Again, no issues whatsoever in years, without complicating the API!

Also, avoiding support for Java arrays is bad for performance when user 
data is in a Java array. Why prevent them from using a critical section 
if that's what they want? JNI offers that, why "remove" this feature?

As for ironing issues with jextract, yes, there are a lot of tricky 
cases. I've started to document what I've typically had to deal with as 
part of the JavaCPP Presets here:
https://github.com/bytedeco/javacpp/wiki/Mapping-Recipes
I'm sure some of the ideas there can help you better design your 
approach even before getting feedback from users. It's only a few 
minutes read!

I'm sorry to hear though that there are still no plans to enhance 
performance where it would matter in most cases, that is by actually 
inlining inline native functions, function-like macros, and C++ 
templates... What would it take for you guys to work on that? I remember 
you said that jextract was not a priority, so why not put that on the 
back burner and work more on the /foundation/ to provide performance 
enhancements that actually matter? (And let guys like me worry about 
parsing stuff from header files.)

In any case, thanks for the binaries. I will be sure to run benchmarks 
to compare the performance against JNI functions. :)

Samuel

On 08/17/2018 06:49 PM, Maurizio Cimadamore wrote:
> Hi Samuel,
> the implementation of the [foreign] branch reflects the document that 
> has been circulating over the last few months:
> 
> http://cr.openjdk.java.net/~mcimadamore/panama/panama-binder-v3.html
> 
> I'm trying to keep that document up to date with developments in the 
> repo, so that it matches the status quo.
> 
> As for the old implementation in the nicl branch, I'm afraid we have no 
> single document that capture its state, other than the list of documents 
> I've shared in the past.
> 
> But some of the things which have improved more markedly in the new 
> branch are:
> 
> * array access - we now have a new Array<T> carrier to model native 
> arrays; the binder turns arrays into instances of this class (and 
> viceversa) - no deep copy required, and works on every type (in nicl, 
> only a handful of types were implemented)
> 
> * function pointers - function pointers now work in all contexts, both 
> as callback arguments to function AND as fields to structs (but we know 
> we need to do more here to control the lifecycle of these things)
> 
> * much deeper macro support; in nicl only string and numeric constants 
> were supported; we now allow arithmetic expression, parenthesis, etc.
> 
> * bitfields are now supported
> 
> * jextract generates correct amount of padding, so that alignment 
> matches that seen by the C compiler at extraction time
> 
> * completely revamped layout language (as per document) backed up by a 
> public layout API
> 
> * public API is fully documented (there's much to do here in order to 
> improve both the API and the doc, but it's a start)
> 
> * many jextract bug fixes to iron out corner cases of the C language
> 
> * the binder has almost been rewritten from scratch; the new code 
> generation model should allow for far better performances (as most 
> operations are achieved by performing an invokeExact on a constant 
> method handle).
> 
> These are the main differences between the two repo and what we have 
> been working over the last few months.
> 
> 
> As to what are we planning to do: we want to keep ironing out issues in 
> jextract and in the binder - we are mostly focusing on 3 areas:
> 
> * lifecycle of function callbacks. The automatic conversion between Java 
> functional interface and function pointers is cool, but seems to magic: 
> it makes the translation non-raw (e.g. setting a function pointer on a 
> struct field requires native memory allocation) - and it is not clear as 
> to when things in memory can be collected (as these objects are 
> synthesized on the fly, in most cases the user doesn't have an hard 
> reference on them).
> 
> * Unresolved layout resolution scheme; as layout can point to other 
> layouts, we need to define more precisely what that means, and what is 
> the context in which this is possible. This is also related to the point 
> below...
> 
> * Significantly redesign jextract to allow for better ease of use; while 
> what we have is good for prototyping purposes, there are too many cases 
> where, in order to get a library working, a lot of fiddling needs to be 
> done, to exclude symbols, or just to have the right set of symbols 
> included in the jextract artifacts. The model of supporting multiple 
> jextract runs is also not clear - e.g. can we extract a library A, then 
> extract a library B and pass the artifacts for A during the second run 
> (e.g. similar to passing a classpath or modulepath to javac) ? We are 
> having some discussion on these topics - and I'll share something more 
> soon. The goal here is to create an experience that is as close as 
> possible to that you get when interacting with standard C/C++ compiler 
> toolchains.
> 
> 
> Once we get some progress going on the tasks mentioned above, our plan 
> is to start the process for releasing early access binaries (Linux/MacOs 
> x64 only, for starters), so that people can play with the bits, without 
> having to build the entire repo from scratch. I expect that this will 
> generate some feedback which will trigger some further iterations on the 
> API/toolchain.
> 
> Thanks
> Maurizio
> 
> 
> 
> 
> 
> 
> On 16/08/18 22:57, Samuel Audet wrote:
>> Congratulations! Is there a changelog or a summary with a list of the 
>> differences? It's hard to keep track of what has been implemented from 
>> just the messages here. Documentation would be great, but I know 
>> there's a lot to be done, so thanks for all the work!
>>
>> Samuel
>>
>> On 08/17/2018 12:46 AM, Maurizio Cimadamore wrote:
>>> Dear citizens of the Isthmus,
>>> as you might have noticed, over the last few months all the 
>>> development activity regarding foreign data/function interface has 
>>> moved from the 'nicl' branch to the 'foreign' branch. We have 
>>> achieved feature parity between the two branches about two months 
>>> ago; right now the set of functionalities available in the foreign 
>>> branch greatly surpasses those provided by the legacy nicl branch, 
>>> which is no longer worked on.
>>>
>>> For this reason, and also to minimize confusion and potential for 
>>> errors, we have just closed the legacy 'nicl' branch. This doesn't 
>>> mean the code has disappeared - you can in fact still update to 
>>> 'nicl' (hg update nicl); but the 'nicl' branch will no longer be 
>>> shown in the list of active branches (by the 'hg branches' command).
>>>
>>> Cheers
>>> Maurizio
>>>
>>
> 