Notes on our current process and forest structure

[Richard Bair]

Hi everyone,

I thought it would be useful to outline our current internal working process, and then describe some of the challenges we need to solve in moving this all out into the open. I'm hoping this sort of insight will help illuminate why everything isn't running smoothly from day one and what we're trying to resolve.

As with the JDK, JavaFX sources are built out of a mercurial forest. The current internal forest structure is basically:

	jfx/			- This is the root forest
	jfx/apps		- Contains a number of public apps (such as Ensemble and DataApp) and internal apps (which have been ported from FX Script to Java but haven't been blessed)
	jfx/glass		- All of the windows / mac / linux / other platform implementations
	jfx/webnode	- WebKit, plus some Java code that glues webkit to the public API
	jfx/media	- GStreamer, plus some Java code that glues gstreamer to the public API
	jfx/tests		- Mostly performance tests
	jfx/runtime	- The bulk of the platform -- all controls, charts, etc are in this repo

There are, basically, four forests:

	master		- The master forest is the forest from which all promotions and releases are built
	graphics		- Used by the graphics, scenegraph, and most other teams
	controls		- Used by the controls team
	media		- Used by the media team

Although mercurial doesn't actually operate this way, you can think of the master as being the root of a tree with graphics, controls, and media as children. Once a week (typically Tuesday) we do integrations. The "integrator" for each child forest will pull the latest from Master into their forest. They will merge up, run tests, run some sample apps, and make sure things are all good to go. They will then push their changesets from their forest (what we call the integration forest or scrum forest) up to the master. The three teams do this according to some negotiated schedule, so that all these merges are done in an orderly fashion. After everybody has integrated, everybody does a fresh pull from master so that after integration, each child forest is in essence sync'd up completely and identical. Then the teams continue working for another week in their own forests, and re-integrate the following Tuesday.

We didn't start with this system in place, at one point we had just a single master. That of course left the master in some undefined state at any given time -- there was no gate to make sure only "good" changes made it to master. So we had an integration and a master forest. But eventually the size of the team became such that we had to split things out further, because changes in one team would be unstable for some few days and this would impact the work of other teams. So we have grown the number of integration forests organically as needed. In general you want as few as you can get away with.

Presently in OpenJFX we have essentially added a fifth integration forest, also called master. Lets call it "openjfx-master". This is actually being populated based on the controls forest. So when the controls folks push fixes into the controls forest, it is propagated to the openjfx-master. This is a short-term position. the openjfx-master is intended to become the real master, and we will fire up additional forests for the controls, graphics, and media scrums to use. At least, that is my first-shot proposal and seems the most sensible since it will mean we continue to work much as we have, and so it will have the least disruption to the team while also opening up the development process.

When somebody has a fix to contribute, they will work with one of these scrum teams (depending on whether it is a controls fix or a media fix or somewhere in the scene graph or core libraries or prism). Fixes will go into the appropriate scrum team forest, and from there will get merged into the master on a weekly basis.

Now, we don't really want to keep forest structure that I described at the start of this message. We really would like to cut down the number of repos in the forest. One proposal is to do something like this:

	jfx/			- This is the root forest
	jfx/apps		- As presently constituted
	jfx/webkit	- WebKit, plus our patches to webkit (but no Java code)
	jfx/gstreamer	- GStreamer, plus our patches (but no Java code)
	jfx/rt			- The bulk of the platform -- all controls, charts, glass, Java parts for web view, java parts for media, performance tests, other tests, etc

In this arrangement we have reduced the number of forests and, if you are only working on the platform, you generally will only need the rt/ and not the full forest (although personally I think everybody should always use the full forest because when doing find-usages in the IDE it is instructive to also have all the apps and samples and such in front of you). So in the process of open sourcing, we're also planning how to make things a bit cleaner and more accessible to developers than they are currently.

In the old closed runtime, we had a bunch of projects such as:

decora-compiler
decora-d3d
decora-d3d-native
decora-es2
decora-jsw
decora-ogl
decora-prism
decora-prism-ps
decora-prism-sw
decora-runtime
decora-sse
decora-sse-native
javafx-anim
javafx-annotation-processor
javafx-beans
javafx-common
javafx-sg-common
javafx-sg-prism
javafx-ui-charts
javafx-ui-common
javafx-ui-controls
javafx-ui-quantum
javafx-ui-webnode
prism-common
prism-d3d
prism-d3d-native
prism-es2
prism-j2d
prism-jogl
prism-null
prism-ps
prism-util

And a lot more. This has grown organically, and over time we've acquired a fair number of projects. Obviously some of these have a relation: all the decora- ones are related with a single decora-runtime and another compiler and then a pile of backend implementations. There is prism-common and prism-util and a pile of prism implementations. Instead of having a flat directory with a huge pile of different projects, we'd like to organize things when we move these to open source. So for example, maybe we do:

Graphics/Decora/
	decora-compiler
	decora-d3d
	decora-d3d-native
	decora-es2
	decora-jsw
	decora-ogl
	decora-prism
	decora-prism-ps
	decora-prism-sw
	decora-runtime
	decora-sse
	decora-sse-native
Graphics/Prism/
	prism-common
	prism-d3d
	prism-d3d-native
	prism-es2
	prism-j2d
	prism-jogl
	prism-null
	prism-ps
	prism-util
Graphics/Glass/
	....
CoreLibs/
	javafx-anim
	javafx-beans
	javafx-common
Controls/
	javafx-ui-controls
	javafx-ui-charts
SceneGraph/
	javafx-ui-common
WebView/
	javafx-ui-webnode

I'm totally making this up, but this is the general idea. Maybe we do this in two steps -- open source first, and then rearrange to clean things up. Or maybe we do it in one go. In any case, I wanted to take a few moments and mention at least somewhat of what we're doing. In addition to doing the due-diligence on third party code checks and making sure header files are all correct and so forth, we're also taking a look at the tools used to build and the structure of the forest to see if there is anything we can do to make it easier for developers to grok. It was painful enough just getting new hires up to speed, with all this in the open and trying to make it accessible to a wide range of new engineers it becomes even more important to make it clean and easy to use.

Cheers
Richard

PS I might be a bit out of date with the exact specifics. I think glass may have already been folded into runtime, and the runtime name is actually not runtime anymore, but I digress.