FINAL PROPOSAL - Java Foreign Function Interface

[Wayne Meissner]

                   Proposal - Java Foreign Function Interface

1.0 Outline

This proposal is for a project to provide Java programmers with easier access
to native libraries.

Although JNI already provides programmers with access to native libraries, it
has some drawbacks:
  a) It requires compilation of a native library for each potential target
     platform.  This places an extra burden on developers.
  b) The complexity of JNI programming is greater than that of a pure java
     project.  It is also usually more complex than the native APIs that the
     programmer wants to call.

For many tasks, being able to access native APIs using java syntax, with
all the hairy details of JNI programming hidden away would be a good thing.


2.0 Need and Community Benefit

Although everyone would like to have 100% pure java libraries and programs,
that is not practical if Java is to compete with e.g. C# as a modern desktop or
systems programming language.  There will always be native APIs that are not
exposed in the current JRE and there will always be a lag between the latest
JREand what the majority are using.

Although JNI can be used to access native APIs, it is somewhat more complex
than either java or the native APIs that programmers are trying to access, and
as such presents a higher barrier of entry than programmers would desire.

Although there is a viewpoint that this barrier encourages development of pure
java libraries, in the case of highly complex native frameworks (e.g. media
libraries), re-implementation in java is impractical.

Another need is from the JRuby and Jython communities - where the ability to
provide a more accurate implementation of the Ruby/Python environment to
programs written in those languages would be eased by the ability to access the
same native libraries that the original implementations of those languages
allow.  The benefit is not just restricted to implementation of JRuby/Jython
features in Java - these languages will also be able to directly call native
functions.

Even within the JDK itself, having easier access to native libraries might
enable faster feature implementation where access to native APIs is needed.


3.0 Concept Viability

The concept of simplified native library is not new - there are several
projects, both commercial and open source.

For example, the JNA project[1] has been used to successfully build bindings
for the gstreamer multimedia framework.  The resulting bindings are pure
java (except for a dependency on JNA itself), and run without modification on
Linux/x86, Linux/amd64, Windows/x86, MacOSX/x86, Solaris(x86, sparc, sparcv9)
and FreeBSD/x86.

The gstreamer-java[2] bindings also demonstrate that it is possible to produce
reasonably performant native bindings using this method - the bindings can
render video into a lightweight swing component at better-than DVD resolution,
with no native code other than the common JNA stub.

The JRuby project is also now using JNA to provide access to native libraries
to Ruby programs so they can run un-altered on the JVM.

Although the idea of providing direct access to native resources from java code
might seem abhorrent to some, it is not without precedent.  The
sun.misc.Unsafe, java.nio.Bits classes already provide the ability to allocate,
read and write native memory directly from java.


4.0 Proposal Details

This project proposes to implement a minimal set of java and JNI components that
facilitate calling native functions directly from Java, and to enable java code
to be called from native code (closures).

Although the project will not include higher level C language features
(e.g. structures, unions, custom mapping between java types and native types)
it will provide sufficient functionality to allow these features to be
implemented by other java libraries without requiring native code.

There are two main focuses of this project - performance, and to provide
sufficient functionality upon which higher-level functionality can be built.

4.1 Dependencies

The proposal depends on libffi - part of gcc, but licensed under a more liberal
license than the GPL - for the low-level native dispatch functionality.

4.2 Supported Function Parameter Types

The project will support primitive java types (byte, short, int, long, float,
double), primitive arrays, and NIO Buffers as function arguments.  All other
java types can be converted to one of these types before being passed to the
native call.

4.3 Native API defined in Java

Instead of using an IDL or XML to define the interface, the project will adopt
the JNA convention of defining the C API in a Java inteface.
e.g.
    public interface libm {
        double pow(double x, double y);
        double log(double x);
    }
    libm libm = Native.load("libm", libm.class);
    double d = libm.pow(2.0, 3.0);

4.4 Expose argument marshalling to code generators

One of the performance drawbacks of existing implementations, is that they
funnel all calls through a reflection proxy - this erases the type of each
parameter, which then has to be inferred again, before the argument can be
correctly marshalled.  It would be nice to allow the type information to be
passed directly to the marshaller, where that type information is already known.
e.g.
    Function f = NativeLibrary.getInstance("m").getFunction("pow");
    double result = JNICall.create(f).addDouble(0.1234d).invokeDouble();

This could benefit both static code generators (e.g. a modified version of
gluegen), or the JRuby/Jython JIT.


5.0 Milestones
5.1 Basic invocation of C functions with primitive java type parameters

5.2 Basic Array and NIO Buffer support.

5.3 ByReference primitive java type parameters - i.e. the equivalent of passing
    the address of a variable to a C function.

5.4 Critical (pinned) array and heap Buffer support.
    Allow programmers to mark array and Buffer arguments to terminal functions
    (i.e. those that won't call the JVM) to allow the JVM to avoid copying
    the heap data to/from native memory.

5.5 Out-only and In-only heap buffer/array optimizations.
    This is an optimization for heap buffers or arrays which cannot be pinned,
    but only need to copy the data one way - either in to native memory, or out
    to java memory.

5.6 Support for String, StringBuffer, StringBuilder parameters.

5.7 Closure support - callbacks from native code to java code.


6.0 Dependence on Sun
There is no dependence on Sun for this project.


7.0 Developer

Wayne Meissner has been a contributor on the JNA[1] project, creating the libffi
backend, porting JNA to 64bit and big endian architectures, and implementing
some higher-level features such as custom type mapping.  He has also created
java language bindings[2] for gstreamer using JNA.


8.0 References

[1] JNA http://jna.dev.java.net
[2] gstreamer-java http://code.google.com/p/gstreamer-java