Refactoring JavaFX Builds & Sources

Fri Feb 1 09:07:12 PST 2013

I just pushed to the rt repo my two "work in progress" gradle scripts, build.gradle and settings.gradle. You will find that these don't work. The reason is that I have another (still closed) "generator" gradle script which takes the existing sources and munges them around into a directory structure that the new gradle scripts work against.

I'm working with the full set of sources and trying to get the new build to work with the full set of sources. I'm targeting the end of the month (Feb) for having the build system sorted out.

For example my "generator" script looks like:

task generate << {
    // Copy over the new gradle build file
    copy {
        from "."
        into "javafx"
        include "build.gradle", "settings.gradle"
    }

    // Create the build-tools project
    copy {
        from "rt/javafx-annotation-processor/src",
             "rt/decora-compiler/src"
        into "javafx/build-tools/src/main/java"
        exclude "META-INF/**", "**/*.properties", "**/*.stg", "**/JSL.g"
    }
    copy {
        from "rt/javafx-annotation-processor/src",
             "rt/decora-compiler/src"
        into "javafx/build-tools/src/main/resources"
        include "META-INF/**", "**/*.properties", "**/*.stg"
    }
    copy {
        from "rt/decora-compiler/src/com/sun/scenario/effect/compiler/JSL.g"
        into "javafx/build-tools/src/main/antlr"
    }
    copy {
        from "rt-closed/javafx-beans/src"
        into "javafx/build-tools/src/main/java"
        include "com/sun/javafx/beans/annotations/**",
                "com/sun/javafx/collections/annotations/**"
    }

…. and lots more
}

You can see some paths are "rt/" and some are "rt-closed/". What I can do is create a generator.gradle file in rt that contains only the portion that is already open so that people in the community can try out the new structure / contribute / etc? The main gradle build file then needs to just include jfxrt.jar on the build path as the binary plug.

Thoughts?
Richard

On Oct 19, 2012, at 7:24 AM, Danno Ferrin <danno.ferrin at shemnon.com> wrote:

> 
> 
> On Fri, Oct 19, 2012 at 7:31 AM, Mario Torre <neugens.limasoftware at gmail.com> wrote:
> > Further more, you should be able to do "find usages" in your IDE and get EVERYTHING IN THE UNIVERSE, rather than right now where in NB it doesn't have enough information about the 160 other projects in our system to be able to do this. This is bad.
> >
> > Right now we build with ant. Many of you have suggested Maven. Some of you at JavaOne suggested Gradle. I am presently looking into Gradle. The key things I like about Gradle are:
> >
> >         1) It isn't XML
> >         2) Like Ivy & Maven, it handles dependencies well
> >
> > We presently have a pile of XML files and nasty build logic for downloading dependencies (this is all in the closed repo, but you would be exposed to the horror if we just made it all public). I wanted instead to use Ivy at the very least, but rather if I can get better build happiness from another setup entirely (like Gradle) then I might as well give it a try.
> 
> I kind of like Gradle, but let me put the cart before the horse here
> (I don't know if this sounds as well in English as it is in Italian,
> where we actually use the oxen, but well :)
> 
> While Maven is a well understood tool, Gradle is fairly new. This
> turns to be especially problematic when building and shipping on Linux
> distributions because there are some constraints and limitation on
> what can go or not in a given release (btw, I think this could be
> considered a problem on Windows and OSX installations in some
> environments where you can't simply install any tool you want).
> 
> Maven, on the other end, is pretty likely sitting on your machine
> already (as are Make, GCC and Configure most of the time on developers
> machines), so I suspect it will be easier for the average folk as well
> as heavily filled packager to take care of the build bits.
> 
> Gradle has a "wrapper" script that can be checked in with the build (the gradlew script), it will download the specific version of gradle and build it in place, from the build. So having Gradle on your machine is not a requirement, the gradle script will build with it in place like it was an external library.
> 
> Plus, Android has just shifted to building user projects with Gradle instead of Ant.  So there will be many Linux distros that will ship and approve Gradle just for that reason.
>  
> 
> Oh my, I would have never thought I would try to sell Maven myself! Of
> course, don't make this stop evolutionary attempts at solving the
> build process, but I thought it's one consideration to take into
> account when deciding what build system to use, since it's such a
> fundamental part of the code.
> 
> Ultimately, the build infrastructure should be what makes more sense
> for the project obviously.
> 
> >         1) I'll make sure cross builds all continue to work
> >         2) Partial builds (for native code particularly) will continue to work such that you don't HAVE to build native code
> >         3) It will produce bit-for-bit compatible output in the artifacts/ directory, so all our RE (Release Engineering) scripts continue to work with little or no modification
> >
> > Also, in my tests so far the speed of the build is dramatically faster -- but then I don't have everything in yet so that might just be wishful thinking. I hope and expect it to be faster though.
> >
> > We generate code in our build system for the following reasons:
> >
> >         1) We have a VersionInfo class which the build system will update with version information. It is a com.sun.something class
> >         2) We have an annotation processor which generates all of the builders
> >         3) We have two grammars that Antlr has process to produce parsers -- one for JSL (for our shaders) and one for CSS
> >         4) We have a "decora compiler" which takes JSL files and feeds it through the JSL parser and produces code / shaders
> >
> > In addition to code generation, we also build a ton of native code (gstreamer, webkit, prism, fonts, glass, image loading). So that has to fit into the build system. Building native code is SLOW, so being able to avoid it for "normal" developers, and being able to avoid native builds when nothing changed in the native code, is important.
> 
> Speaking about this, I wonder if we will end up doing like IcedTea
> with OpenJDK, since most of those probably have an OS counterpart we
> would rather want to use (in other words, make those elements somehow
> pluggable).
> 
> > I have found that some of our unit tests are written such that they require more of the platform, rather than just one class. So for example, there are tests in the scene graph which might want to use a UI control for some purpose. I'm thinking the easiest thing to do is to have a test directory at the top level which houses all the tests, and break them up based on whether they are part of the smoke test, integration test, or manual test suite. Smoke tests are those that are run continuously. They must be headless, and they must be completely automated, and they must execute within some reasonable amount of time (since hudson is going to be doing it continuously). The integration tests are the rest of the automated tests that need to be run between every integration with "master". Finally the manual tests are not run as part of the integration / smoke tests, but really are just little "toys" or apps we've written that are useful when developing a feature or whatnot. We've got a ton of these little guys (mostly of the "HelloWorld" variety), and these would go into "manual".
> 
> This is one advantage of maven, in that you are somehow forced to have
> this layout. Maven supports the idea of modules, they are a hierarchy
> that pretty much resembles what the project structure you depict later
> looks, and for each of those modules you would have unit tests.
> Integration tests could simply go into an high level module, the same
> for the manual tests.
> 
> The problem with Maven is when you get off the beaten path (like running your own annotation processor, shipping source code in the jar, custom build languages) the build script gets very verbose and hard to read.  All the XML creates more noise.  Yes, you can do it, but it becomes difficult to maintain.  Build scripts for Maven projects that follow the maven conventions are dead simple, but OpenJFX is not one of those run of the mill builds.
> 
>  
> 
> > We also will have an apps directory, where we will place all the samples (like Ensemble) and experiments (like the JavaOne Schedule Builder).
> >
> > And I want to have a "benchmarks" directory where all of our benchmarks will live. The only problem with this one is that we have a dependency on JRockit benchmark harness, and I'm not sure how to resolve that with the open source bits. Maybe the JRockit benchmark harness is already open source, I haven't dug into it yet to find out. However I believe we *must* have benchmarks open, as well as the SQE tests. My goal is to run OpenJFX in the open and to have meaningful contributions -- how can that happen if we're keeping back essential verification tools? How can somebody submit a patch and not know whether it is going to impact performance? So I want to get the benchmarks all out, but we need to figure out what to do about this dependency. Another option is to replace this dependency with one of the other open source benchmark harnesses, but this would need to be carefully considered as it would disrupt our benchmark reporting for some period of time. But ultimately having public numbers and public tests would, I think, be very beneficial across the board.
> 
> As a side note, we're developing a monitoring tool for OpenJDK
> (Thermostat). I don't want to put Thermostat in competition with
> JRockit (well, I did so already!), but by the time the full code is
> out, we will be quite likely ready for prime time. There are also
> other tools that do pretty good job, like VisualVM and JConsole.
> Depending on the harness you have (which I assume means loads of
> binary files with dumps and performance related data), we can either
> try to make them compatible with all those tools, or better yet,
> develop some kind exchange format for Performance Analysis (so that we
> are not really depending on any tool). This can be a cool
> collaboration between some other teams (I suspect such an effort would
> go beyond JavaFX of course). In any case, having the harness in the
> code doesn't hurt (as soon as we don't depend on them to build the
> final code).
> 
> Cheers,
> Mario
> 
> --
> pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
> Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF
> 
> IcedRobot: www.icedrobot.org
> Proud GNU Classpath developer: http://www.classpath.org/
> Read About us at: http://planet.classpath.org
> OpenJDK: http://openjdk.java.net/projects/caciocavallo/
> 
> Please, support open standards:
> http://endsoftpatents.org/
> 
> 
> 
> -- 
> There is nothing that will hold me back.  I know who I am....
> I remember wher I came from, and I feel stronger for knowing.
> Zane, Ninja of Ice.  Ninjago S01E07
> 