Porting JavaFX

[Richard Bair]

The post is intended to lay out the different activities that need to be undertaken in order to do a port from JavaFX to any platform, but with special emphasis on iOS. There are many, many details which will only be discoverable and actionable once the code is all open sourced, so this post itself is necessarily only high level and intended as some kind of guide to ensuing discussions.

There are, essentially, 3 things that must be done to port JavaFX to any platform:
  * Port a JVM and the Java class libraries
  * Port JavaFX (Glass, and to some minimal extent, Prism)
  * Provide a toolchain for building apps for the target environment

In the existing desktop ports, #3 is sort of taken for granted because you have javac and such available to you. However as you attempt to target app stores, the jfxpackager & app bundling becomes much more important. On iOS, there is a lot here that has to be done, providing "entitlements", code signing with an Apple certificate, proper bundle generation, etc. There will most likely need to be some toolchain interop with Xcode for example.

*Port A JVM and the Java Class Libraries*

This is the first major body of work. Getting a JVM that will run on the target device is the first order of business. There are several options available, although no official HotSpot support (yet) for iOS. Some open source VM options might include XMLVM or Avian. Any VM that wants to have acceptable performance on iOS and WinRT will need to support ahead-of-time (AOT) compilation since just-in-time (JIT) compilation is prohibited on these platforms. [Aside: on iOS for example it simply *is not possible* to mark a memory segment as executable. That means that there is no way to have runtime code generation. This is done for security reasons -- JIT security bugs in fact do exist, the Safari JIT having had such bug publicly exposed a year or so ago. Note that Apple allows Safari to JIT, but they aren't going to let 3rd parties do the same). 

The next step is getting a port of the Java class libraries. Some things will just work -- anything that the JVM was responsible for implementing, such as threading, or anything that is just plain Java code. But many other things such as networking and imageIO, for example, will need to be implemented. For iOS, starting from the OpenJDK Mac code base might be a good start. You don't have to worry about Swing / AWT since the basis we would use to target these devices is JavaSE Embedded, not JavaSE itself, and I do not believe that Swing/AWT are required for SE Embedded. But if you are really adventurous, there is no reason (that I know of) that you couldn't get all of JavaSE working since the Mac OS port of SE uses a lightweight AWT (AWT components are emulated using Swing -- let that swirl in your noodle for a while ;-)).

For a quick and dirty hack, you can cut all kinds of corners here. For an actual GA release, porting all the required SE libraries over is an absolute necessity. This is, in my estimation, the most difficult part of doing a port to iOS, but maybe that is just because I'm not as familiar with what needs porting on SE vs. FX.

*Port JavaFX*

Porting JavaFX to a platform mostly involves a new port of Glass. On all ports that we have now (and most of them that we've prototyped), we reuse Prism and Glass and simply provide a new port of Glass. At one point in the past, when Prism & Glass were still not GA quality, we maintained a completely separate Toolkit implementation (com.sun.javafx.tk.Toolkit) which was based on Swing / AWT. This level of abstraction we want to remove from the platform, such that the Scene Graph classes (what I often call "the top half" -- meaning, the top part of our cake diagram) will refer directly to Glass / Prism classes ("the bottom half"), whereas now there are a series of interfaces that separate the two.

At this point, every port we're looking at will be a port of Glass / Prism. Most of the work is in Glass. Glass is the "windowing layer", and has all the code for handling the event queue, windows, views (areas inside the window that get rendered), etc. There is a boundary between Prism & Glass that is kind of messy which is responsible for negotiating the creation of the View surface so that it is established correctly for OpenGL. Otherwise Prism pretty much just works, because we use OpenGL ES 2 for both desktop and mobile, and we have a pretty good implementation that we use everywhere else.

Of course, you could also provide a stylesheet that gave native looks to iOS controls (not hard to do), and would be quite a nice addition to the platform (now that Jasper has added public API for setting the user agent stylesheet).

Right now, Glass is designed such that there is a non-trivial amount of native code involved in doing a port. Anybody with experience with Objective-C and iOS should feel reasonably well at home in it. Ideally we'd use a heck of a lot less native code to do a port, but for now, what we have is what we have.

*Toolchain*

I've about mentioned as much as I wanted to mention on toolchain. It is an important part of the story, so as to make developers productive. Ideally we want to allow people to develop on any platform for any platform. Some work will need to be done to see if that is even possible for iOS. Today every toolchain I know of for iOS requires Xcode. Maybe you can figure out how to do the cryptographic signing from Linux or Windows instead, and not require Xcode at all.

Also, of course, creating a co-bundled application using a subset of the JRE is critical to get sizes down to something respectable. I believe 50mb is the limit for an application that wants to be downloadable over-the-air. Obviously we want to keep the runtime to a reasonable fraction of that to allow room for the application itself.

*Performance*

On iOS, our existing prototypes don't have AOT. This has proven to be a big barrier to performance. I actually was blown away at how fast the hotspot interpreter was on iOS -- but no matter how fast it was, it is really (*really*) hard to get smooth graphics performance without AOT. We did do an amazing amount of performance work in this area a few months ago which has benefited desktop & embedded as well (often to a surprising degree). However there is only so much you can do in an interpreter.

On the other hand, AOT, we believe, when used on everything will balloon the size of the application. So AOT needs to have some kind of control of what is AOT'd. Critical sections of Prism and the Scene Graph would need to be AOT'd to get optimal performance. We had an example where we ran FX on normal iPhone and on a jail broken iPod with JIT, and it was very fast on the iPod with JIT. We got to the point where the interpreted JVM on iPhone was quite good even for the schedule builder (which is reasonably complicated) -- the performance (on interpreter!) was indistinguishable from native, but required various performance tweaks and hacks in the application code.

So although without a JIT with FX we could get good performance with effort on the application developer's part (requiring some extensive knowledge of the platform), with AOT, performance is good by default.

Android was interesting. We had two ports, one that used the Android graphics APIs and one that used Prism / Glass. The Prism/ Glass version was at least as fast as the one using Android APIs. This validated our notion of using Glass / Prism everywhere and removing the Toolkit APIs as it was our feeling that we would not any longer need this level of abstraction.

*GWT Port*

For what it is worth, a few years ago we prototyped several ports of JavaFX using GWT to be able to run in a web browser without a plugin. This is not likely to be something that *we* will do, because any non-plugin solution for the web will not be "real" Java -- it cannot properly support threading and many other things that are integral to a real WORA solution around Java. Some solutions such as Emscripten are very interesting to me, which basically is a VM that JIT's into JavaScript (!!). I've long dreamt about having a JVM implemented in JavaScript such that we could run real Java in the browser without a plugin. Of course, performance is generally abysmal because you are interpreting Java in JavaScript. However with something like Emscripten, if we were very, very clever, you might get "browser native" performance by translating Java byte codes into JavaScript code which would then be handled natively by the browser. Key problems: threading & memory management (missing PhantomReference, WeakReference, etc support in JavaScript).

However, even if *we* are not going to support GWT-like solution, there is no reason that the open source community could not do so.

Our solution for this explored a few options. The idea was that much of the public API would be able to translate directly to JavaScript via GWT. Also the idea was that the Toolkit interface allowed us to virtualize the threading rules, such that you could have an implementation where the UI thread and render thread were the same thing, or different things. And in fact we will try to preserve this as we remove the Toolkit abstraction, such that a JavaME port (for example) or a GWT port could continue to work. Since there is not good threading support in JavaScript, we need all of the FX code to run in the same thread.

There would no doubt need to be patches to enable this kind of implementation, and I'd be view such attempts very favorably.

We had an implementation that used SVG for rendering, an implementation using DOM + Canvas, and DOM + WebGL. There were problems with all attempts, but I think each worked reasonably well for many things. Typical problem areas are effects, editable text, and media. If you could run most of Glass / Prism then using WebGL at the very bottom might work, but performance isn't going to be all that grand. Performance of animations is going to be problematic unless you can leverage CSS animations in some way. There are a lot of issues here.

That's it for now.
Richard