Refactoring JavaFX Builds & Sources

[Mark Fortner]

Hi Richard,
There's probably two overlapping aims here: 1) to build JavaFX itself, and
2) to build JavaFX projects (in your case, projects like Ensemble).

As you pointed out, Maven can be rigid if you want to do something that it
doesn't normally do.  It's goal is value convention over configuration.
 But you can mix and match Maven and Gradle, or you can write your own Mojo
to provide any custom functionality (it's similar to writing an Ant Task).
 The nice thing about Maven is that you can use the artifacts with Gradle,
Ivy and other build tools.  And those tools include plugins that let them
interoperate with maven builds.

As I mentioned in this issue: http://javafx-jira.kenai.com/browse/RT-25723
You can generate project files for most IDEs.

Most external developers won't really want to build the native bits.  In
the past I've created jars that include native DLLs, SOs, and added those
as dependencies.

If you are using Hudson or some other CI tool, it should be able to handle
nightly builds and publish them to a Maven compliant repo (thus making them
usable by anyone using Maven, Gradle, or Ivy).  The next time a developer
builds, it will fetch the latest build including the native bits (assuming
that they've configured their POMs to depend on the latest SNAPSHOT).
 Hudson can also handle generating the coverage, and test reports for you.

Mark


On Thu, Oct 18, 2012 at 12:16 PM, Richard Bair <richard.bair at oracle.com>wrote:

> I've spent a couple days (spread out over the last week and a half or so)
> looking at our build system and what we can do to improve it. As I have
> mentioned in other threads, my test for success in the build system is if a
> developer wanting to change Java code simply had to:
>
>         1) Download (or Clone) the sources
>         2) Open the JavaFX project in their IDE of choice (IntelliJ IDEA,
> Eclipse, Netbeans)
>         3) Hit "run" and it runs Ensemble.
>
> And likewise, hitting "test" should run all the automatic tests and
> hitting "coverage" should generate code coverage stats, and "debug" should
> work -- exactly as one might hope. Today, in order to make this work, Steve
> has a carefully constructed Eclipse project setup and I have a carefully
> (laboriously) constructed IDEA setup and nobody has a very good NetBeans
> setup. This is bad.
>
> Further more, you should be able to do "find usages" in your IDE and get
> EVERYTHING IN THE UNIVERSE, rather than right now where in NB it doesn't
> have enough information about the 160 other projects in our system to be
> able to do this. This is bad.
>
> Right now we build with ant. Many of you have suggested Maven. Some of you
> at JavaOne suggested Gradle. I am presently looking into Gradle. The key
> things I like about Gradle are:
>
>         1) It isn't XML
>         2) Like Ivy & Maven, it handles dependencies well
>
> We presently have a pile of XML files and nasty build logic for
> downloading dependencies (this is all in the closed repo, but you would be
> exposed to the horror if we just made it all public). I wanted instead to
> use Ivy at the very least, but rather if I can get better build happiness
> from another setup entirely (like Gradle) then I might as well give it a
> try.
>
> Of course, I'm hoping Danno & Andres will help maintain the build scripts
> :-D
>
> I have two different scripts, one which is to be used during transition,
> and will generate the "new world" from the old, and another which will be
> the build file going forward. I want to emphasize that this is at the
> moment only a prototype. Also, there will be no loss in functionality
> migrating from the current system to any future system. In particular:
>
>         1) I'll make sure cross builds all continue to work
>         2) Partial builds (for native code particularly) will continue to
> work such that you don't HAVE to build native code
>         3) It will produce bit-for-bit compatible output in the artifacts/
> directory, so all our RE (Release Engineering) scripts continue to work
> with little or no modification
>
> Also, in my tests so far the speed of the build is dramatically faster --
> but then I don't have everything in yet so that might just be wishful
> thinking. I hope and expect it to be faster though.
>
> We generate code in our build system for the following reasons:
>
>         1) We have a VersionInfo class which the build system will update
> with version information. It is a com.sun.something class
>         2) We have an annotation processor which generates all of the
> builders
>         3) We have two grammars that Antlr has process to produce parsers
> -- one for JSL (for our shaders) and one for CSS
>         4) We have a "decora compiler" which takes JSL files and feeds it
> through the JSL parser and produces code / shaders
>
> In addition to code generation, we also build a ton of native code
> (gstreamer, webkit, prism, fonts, glass, image loading). So that has to fit
> into the build system. Building native code is SLOW, so being able to avoid
> it for "normal" developers, and being able to avoid native builds when
> nothing changed in the native code, is important.
>
> Also, testing. We have a variety of tests that we need to organize better.
> At the moment we have basically two types of tests: JUnit tests and SQE has
> visual tests (using Jemmy). The unit tests are all going open source along
> with the code projects they belong to. We have not yet got complete
> clearance for all of the SQE tests, although I firmly believe it is the
> right thing to do, and will push on it again once the rest of our
> "homework" (fixing the build system etc) is taken care of.
>
> I have found that some of our unit tests are written such that they
> require more of the platform, rather than just one class. So for example,
> there are tests in the scene graph which might want to use a UI control for
> some purpose. I'm thinking the easiest thing to do is to have a test
> directory at the top level which houses all the tests, and break them up
> based on whether they are part of the smoke test, integration test, or
> manual test suite. Smoke tests are those that are run continuously. They
> must be headless, and they must be completely automated, and they must
> execute within some reasonable amount of time (since hudson is going to be
> doing it continuously). The integration tests are the rest of the automated
> tests that need to be run between every integration with "master". Finally
> the manual tests are not run as part of the integration / smoke tests, but
> really are just little "toys" or apps we've written that are useful when
> developing a feature or whatnot. We've got a ton of these little guys
> (mostly of the "HelloWorld" variety), and these would go into "manual".
>
> We also will have an apps directory, where we will place all the samples
> (like Ensemble) and experiments (like the JavaOne Schedule Builder).
>
> And I want to have a "benchmarks" directory where all of our benchmarks
> will live. The only problem with this one is that we have a dependency on
> JRockit benchmark harness, and I'm not sure how to resolve that with the
> open source bits. Maybe the JRockit benchmark harness is already open
> source, I haven't dug into it yet to find out. However I believe we *must*
> have benchmarks open, as well as the SQE tests. My goal is to run OpenJFX
> in the open and to have meaningful contributions -- how can that happen if
> we're keeping back essential verification tools? How can somebody submit a
> patch and not know whether it is going to impact performance? So I want to
> get the benchmarks all out, but we need to figure out what to do about this
> dependency. Another option is to replace this dependency with one of the
> other open source benchmark harnesses, but this would need to be carefully
> considered as it would disrupt our benchmark reporting for some period of
> time. But ultimately having public numbers and public tests would, I think,
> be very beneficial across the board.
>
> So, I am proposing a project structure something like:
>
> javafx/
>         apps
>                 experiments
>                 samples
>         build-tools
>                 src/java (annotation stuff)
>         modules
>                 base (javafx.beans, javafx.collections, etc)
>                 graphics (scene graph, prism, glass)
>                 controls (controls, charts)
>                         src/
>                                 decora-d3d
>                                 prism-egl
>                                 ...
>                                 java
>                 swing (javafx.embed.swing)
>                 swt (javafx.embed.swt)
>                 media
>                 web
>         tests
>                 smoke
>                 integration
>                 manual
>
> Each module directory is basically just a home for the sources. All native
> sources are right there along with the Java sources, and we pick the right
> natives to use depending on what the build target is (for all the
> cross-compile stuff).
>
> Thoughts?
> Richard