Panama Updates

[John Rose]

Hi Tobi.

On Dec 10, 2015, at 8:43 AM, Tobi Ajila <atobia at ca.ibm.com> wrote:
> 
> Based on discussions at JavaOne, it sounds like you (Oracle) have some
> interesting results / updates on Panama.  Can you share your progress on
> this list?
> 
> In terms of what we've been doing, our initial Layout runtime prototype is
> now available on github, along with some design docs.  Have you had a
> chance to take a look at this?
> 
> Link to prototype: https://github.com/J9Java/panama-layout-prototype
> Link to docs: http://j9java.github.io/panama-docs
> 
> Regards,
> —Tobi

Thanks for the links to your works-in-progress.
I've downloaded your prototype code and verified that it
passes tests.

For us, it is all implementation and not much spec. so far,
which is one reason why we've been quiet on this channel.
(We wish we could share impl. code with you, but we
understand it's not practical at present.)

We have been experimenting furiously with the groveller
(called "jextract") and also with native function binding.
Some of our work had already been posted to the
Panama project repository in OpenJDK.  There is more
to come, probably early next month, as we polish work
done to get ready for JavaOne.

We are able to extract, bind, and run some of the canonical
examples:  getpid, printf (varargs), qsort (callbacks).

We can also run jextract using its own output for extracting
bindings for the libclang API.

The key spec. content related to jextract, of course,
is defining the annotations and utility types present
in the extracted JAR, along with rules for mapping native
"source types" and "source names" to valid Java "carrier
types" and "carrier names", plus rules about the structure
of the interfaces and methods that embody the extracted API.

A year-old prospectus of this spec. is here:
  http://cr.openjdk.java.net/~jrose/panama/metadata.html
This needs to be updated, of course, to reflect what
jextract currently puts out, and where our current
thinking is leading us.

I have posted a couple of diagrams sketching the tool
chain we are looking at:
  http://cr.openjdk.java.net/~jrose/panama/Metadata-Flow-User-View.pdf
  http://cr.openjdk.java.net/~jrose/panama/Metadata-Flow-Possibilities.pdf
The second one is much larger than the first, and is
a range of possibilities, rather than a road map.

In your prototype you have @LayoutDesc to annotate
carrier types with their carrier names and types (or those
of their components).  This is very similar.

For data structure, jextract stuffs data about original source types
and source types in annotations on the extracted "carrier APIs".
For functions, jextract generates interface methods
whose descriptors have the corresponding carrier types,
and adds annotations if necessary.  It's all dumped into
the JAR produced by the groveller.  Henry Jen has been
doing this work.

The Panama runtime loads these interfaces, reads the annotations,
and binds private implementations to them using Unsafe, etc.
by spinning bytecodes for hidden classes.  Vladimir Ivanov
wrote JVM binding logic which uses a "linkToNative" primitive
that wraps a method handle around a name entry point and
associated native type descriptor.  An year-old prospectus
of this binder is here:
  http://cr.openjdk.java.net/~jrose/panama/native-call-primitive.html

Recently, Mikael Vidstedt has been experimenting with lightweight
mechanisms for specifying the bindings of function call arguments
and return values, and binding them quickly at runtime.  He has
a way of flexibly mapping an ABI to a wide set of machine types,
including multi-lane vectors (on X86).  This notation, which is
a little bit like a layout description, might possibly appear in
groveller output, although it is likely that it can remain internal,
since the function signatures are not ambiguous.

Mikael is working on a presentation about Panama status,
and he allowed me to post a version of his work in progress:
  http://cr.openjdk.java.net/~jrose/panama/panama-status-2015-1216.pdf

Vladimir is experimenting with handling types larger
than 64 bits from Java.  The basic building block is a type
called Long2 (or Long4 or Long8) which carries a tuple of
longs, without assigning them any semantics.  These types
allow us to work with APIs that work on vector types,
including APIs like intrinsic functions in <immintrin.h>.

I had an intense lunchtime conversation with Dan Heidinga
about the LDL.  I will post more thoughts about this separately.

Best wishes,
— John