Modularisation and repositories (forked from Re: Building JavaDoc and Sources JARs)

[Richard Bair]

Thank you for this explanation, it was very easy to follow!

> (As an aside, with Maven I never check in my intellij project files, and tend to get my developers to open the POM file directly - the POM file becomes the source of truth and is IDE agnostic). 

Ideally we'd do the same with Gradle, although the IDE's do not yet have as good native support for it as they do for maven (although work seems to be happening). Also, if we had a binary repo setup from which we could draw for builds, it would probably make the tools happier. The main issue we run into is that we have to manipulate the ext class path because the JDK you download has an "old" jfxrt.jar in the ext class path (which is after the boot class path but still before your app), so we have to use a -D flag to remove it from there so that we can make sure we're building against our own code instead of the older jfxrt.jar that shipped with whatever JDK you have installed.

> I don't really know how your internal build works, how often it happens, etc, and how you structure milestones.

We use Hudson. We do continuous builds on all forests (graphics, controls, media, deploy, master). I believe nightly we run many of the unit tests (what we call smoke tests), but it might be more often. Each week we sync up between all the child forests and master, produce a build on master, and pass this to the JDK. The JDK then includes these build artifacts in the weekly JDK promoted build.

The OpenJDK milestones page (http://openjdk.java.net/projects/jdk8/milestones) lists the official milestones. If we were more agile, these would be treated as 'mini releases' where the bug counts and burn downs and such are part of each milestone planning. The dates look like they were anywhere from 2-6 months between milestones. I don't know what the plan is for JDK 9 milestones.

Within JavaFX, teams use a variety of planning methods. We refer to "scrum" but we don't follow scrum exactly. In fact some teams continually experiment on the agile process to find what they like best. Some teams have 1 week iterations, some 2 weeks. Some use JIRA tags to help identify what issues are planned for a given iteration.

The JDK team is working on consolidating and improving our build infrastructure (for all of the JDK not just JavaFX). Our idea is that instead of pushing patches directly into a repo, you submit them to a tool that will automatically do the integration work -- update from master, apply the patch, build & test. If the build & test pass, push the fix to master, update the bug, etc. If it fails the build & test, reject the patch (send email to dev, update JIRA, whatever) and then start processing the next patch(es).

Our tests are to be separated into several "levels", of increasing complexity / time. So for example, our "level 1" tests would include all the automated unit tests, whereas "level 2" tests might include automated GUI tests, etc. The "level 1" tests would execute as quickly as possible while providing as broad coverage as possible; "level 2" would catch additional issues that first set of tests didn't, but necessarily take longer to execute, and so forth. We also have stress tests which have to run for 2 weeks, and a pile of different configurations that everything has to be executed on that takes a long time to complete, etc.

So with this infrastructure we will continue to have continuous builds. One of these will be stamped as a nightly build. The nightly build has by definition passed some level of testing (level 1) so there is enough confidence in it to promote it as a nightly build. The nightly build is then subjected to additional testing. Each week, the most recent nightly build which passes the more rigorous nightly testing (might take 8+ hours to complete) will be promoted as the weekly build. These are further submitted to more rigorous testing (such as stress testing) and bugs found there are fed back into JIRA so we can fix those.

> My strategy here would likely be to have the automated nightly builds deploy a SNAPSHOT release (say 8.0.1.SNAPSHOT) which is always the latest and greatest of 8.0.1. I would then also have these automated builds deploy the exact same code as a versioned build - probably something like 8.0.1-2013-07-29 where the last bit is the date it was deployed. Note that when I say 'deploy' here, I mean into your own self-hosted Maven repo (Artifactory, or whatever you want). You wouldn't deploy nightly builds into Maven Central. 

The following discussion assumes that the worst case is where every push by every other developer breaks my build and causes me to have to update my repo in order to build again (such as if API between Scene Graph and Controls were to change such that I, as a controls developer, have to update my copy of controls sources in order to continue to build).

This worst case scenario might not be that uncommon. I might add some new private com.sun whatever API and then update some code in Controls to refer to it.

Some possible failure modes:
	- Suppose SNAPSHOT means, latest continuous build
		- In the worst case, I have to sync several times during the day
	- Suppose SNAPSHOT means, nightly build
		- In the worst case I have to sync up daily
		- However I can break in the opposite way -- I just did a fresh update / clone, and my sources are more up to date than the nightly build!

Or put another way:

double likelihoodOfBreak = .5;
if (SNAPSHOT == continuous) {
   if (haveOldRepo && I_BUILD) throw new BrokenBuild();
   if (haveNewRepo && I_BUILD) return success;
   if (haveSlightlyOutOfDateRepo && I_BUILD && Math.random() > likelihoodOfBreak) hrow new BrokenBuild();
   return success;
} else if (SNAPSHOT == nightly) {
   if (haveOldRepo && I_BUILD) throw new BrokenBuild();
   if (haveNewRepo && I_BUILD && Math.random() < likelihoodOfBreak) throw new BrokenBuild();
   if (haveRepoFromFirstThingInMorning && I_BUILD) return success;
}

Where the "likelihoodOfBreak" is different depending on what the modules downloaded from the binary repo are. If they are per component (graphics, base, etc) then the likelihood of a break is larger (double value would be smaller) because there are often changes in Java code between components. If the modules are just binary modules, then the likelihood is smaller (double is larger) because we don't alter the Java->Native boundary as frequently (although it still happens with regularity).

Having SNAPSHOT mean continuous builds would mean you always work with the latest and greatest but if you are slightly out of date you may probably break, and if you are wildly out of date you're definitely going to break. Having SNAPSHOT mean a nightly build would mean that the only repo that is guaranteed to work is the one that was checked out with the same tag as that which produced the nightly.

I like the idea of building all java code all the time -- it at least simplifies the problem. For native binaries, what if any time anybody makes a change to the java->native interface, they have to manually rev the version number of that "module"? Then we would be setup to always build against the latest and greatest, and what is the latest and greatest is rev'd each time it changes, rather than each time a build is produced? If you had a general automatic mechanism for that then you could also apply it for Java code if you wanted to.

> The second option is more explicit. Just change the dependencies in your POMs between modules to reference a specific version. So if you are on the controls team and you need to hook into a specific version of base, you just edit the controls POM and where it references base as a dependency put in the version number for the day you want to pin to, e.g. version 8.0.1-2013-07-29. You are then guaranteed to be fixed against that version and never have to worry about a change being forced on you.

I think this is a good solution to the problem of long-out-of-date workspaces. You can always manually set the version to pin to. I think this is better than having a stale bunch of dependencies that you have to manually update.

Richard