Modularisation and repositories (forked from Re: Building JavaDoc and Sources JARs)

[Tom Eugelink]

Chiming in; I indeed would recommend sticking to a version naming scheme that is compatible with Maven, like Daniel says. But there is an behavior in Maven snapshots that may be of interest.

In Maven a snapshot can be updated as often as you like, but a release can be generated only once, so a 1.0 can only be placed in a repository once (or at least not without manual intervention). A release also is never supposed to be removed from a repository. A release can be a 1.0 or 1.0-M12, anything not ending in -SNAPSHOT. So, like Daniel described, using a date also is a release and the artifact is never removed from a repository. That may be a problem, although from a diskspace / management point it hardly would be.

SNAPSHOT releases to a repository manager are handled differently. When releasing a 1.0-SNAPSHOT, it actually is stored with a date time tag, so something like 1.0-20130731-125223.jar. A reference to 1.0-SNAPSHOT refers to the latest snapshot, but one could also refer to a specific version using the datetime tag. Problem is that usually the repo manager is setup to automatically remove older versions (keep the last 3 or keep the last week or so).

But whatever you chose, Maven version scheme  is kinda the defacto standard and sticking to something compatible would be a good thing.

Tom

On 2013-07-29 15:58, Daniel Zwolenski wrote:
> And one other thing I forgot to mention on this topic, the practice of
> having platform specific jars that contain both the non-platform specific
> stuff (usually 90%+) and the platform specific stuff does not fit that well
> into Maven repo deployment.
>
> Maven would prefer to have something like
>
>     - jfxrt.jar - contains all the non-platform specific code
>     - jfxrt-win.jar - only contains the additional bits for windows
>     - jfxrt-osx.jar - only contains the additional bits for OSx
>     - etc
>
> You would then deploy these all under net.java.openjfx:jfxrt:8.0.1 but the
> main jar would have no classifier and the other jars would have a
> classifier for their platform. The source (and javadoc) however would be
> one zip for all them (deployed under the same coordinates but with the
> classifier 'source' or 'javadoc').
>
> Currently I'm stuck in the deployment of the 78 backport as Sonatype won't
> let me close a release without a non-classified JAR but we have only
> operating specific ones that I am deploying with a classifier. I'm waiting
> to hear from Sonatype what they recommend in this case,
>
>
>
>
>
> On Mon, Jul 29, 2013 at 11:34 PM, Daniel Zwolenski <zonski at gmail.com> wrote:
>
>> On Fri, Jul 26, 2013 at 3:47 PM, Richard Bair <richard.bair at oracle.com>wrote:
>>
>> I'm assuming here you're talking about publishing real builds (at least
>>> OpenJFX ones) and not on a local developers machine ('cause there'd be no
>>> advantage to that alone). But maybe you can help me understand another part
>>> of this problem, which is that suppose we have two developers, A and B. A
>>> is on some code two weeks old. B is completely up to date. B does some fix
>>> and pushes it. The build server builds the artifacts and puts them in the
>>> repo. The next time A does a build, it grabs the latest built artifacts for
>>> the code A isn't building (WebView, for instance) and there is a
>>> compile/link error because the new binaries from B are out of sync with the
>>> 2 week old code that A is building with.
>>>
>>> Normally you version for things like this, but in this case we're talking
>>> about shared libraries that are unversioned -- they're SNAPSHOT. But one
>>> snapshot is not equal to another. How to handle this? Right now in the
>>> closed builds we have an explicit "ant update" step you have to run to get
>>> the latest binaries.
>>>
>> I've had similar situations to what you're facing, where I might have a
>> number of core/utility libraries in an organisation and then lots of
>> projects that use these. If one team changes the utilities but the other
>> team doesn't want those changes, it gets messy.
>>
>> I'll give you the overview of solutions I've come up to this and you can
>> take from it what you like. I haven't used Gradle though so I'll have to
>> talk in terms of Maven. Since Maven is more restrictive than Gradle you can
>> hopefully extract what you need. Others might be able to chime in more or
>> different opinions (I'd love that personally).
>>
>> In Maven, you would have each of your modules as a separate Maven module,
>> with it's own POM and it's own coordinates (where it is deployed to in the
>> repo). e.g. using groupId:artifactId:version you would have stuff like:
>>
>>     - net.java.openjfx:openjfx-base:8.0.1
>>     - net.java.openjfx:openjfx-controls:8.0.1
>>     - etc
>>
>> Each of the native modules would also be their own modules with their own
>> POM files and coordinates, so you would have:
>>
>>     - net.java.openjfx:jsl-decora:8.0.1
>>     - net.java.openjfx:jsl-prism:8.0.1
>>     - net.java.openjfx:native-font:8.0.1
>>     - etc
>>
>> Maven typically works best when you group these in a folder hierarchy,
>> with parent POMs as needed. Gradle is more flexible, but I suspect it would
>> still benefit from a the standard hierarchy here, this could look something
>> like (cut down, just indicative):
>>
>>     - openjfx
>>     - base
>>        - controls
>>        - graphics
>>           - graphics-core
>>           - jsl-decora
>>           - jsl-prism
>>
>>
>> Each of those directories would have it's own POM and in the leaf cases a
>> src/main/java (or native equivalent such as src/main/c++). I've introduced
>> graphics-core, since currently graphics is one big blob of Java and native
>> artifacts, making it hard to work with and deploy, etc. It works better if
>> they are separate and in Maven a parent module (such as graphics) would not
>> typically have code in it as well.
>>
>> This directory structure allows you to build each module, or group of
>> modules stand alone. You could go into the controls directory and run 'mvn
>> clean install' and it would build only that, or into the graphics directory
>> and build only the graphics child modules. The top level openjfx module
>> would have no source code but would just provide an uber parent where you
>> could just build all the sub-modules in one command mvn clean install
>> (maybe adding a -Pwin64 to trigger profiles to build the OS specific
>> versions).
>>
>> Each of these modules would then be deployed to its own unique coordinates
>> in your Maven repo (your self-hosted artifactory or whatever). So
>> jsl-decora is in its own directory in the repo and is then referenced as a
>> dependency by graphics-core (or whatever needs to use it). Additionally the
>> JavaDoc and source code for each module would be deployed with each. Each
>> module is a totally stand-alone deliverable - even if it only exists to be
>> used inside a bigger module.
>>
>> This is where it gets nice, since if jsl-decora is available in a repo I
>> have access to, I never need to build it, Maven will just pick up that one.
>> Similar for all the pure Java modules as well - if I just want to build
>> 'controls' but not 'base', Maven will pick up base from the repo. This is
>> good modularisation - you have this on the code level now but not on your
>> build level as far as I can see.
>>
>> Additionally you can open an individual POM file in IntelliJ or Eclipse as
>> its own project. So you could open just the 'controls' project and it
>> should be able to build and run the unit tests, etc, of just that module.
>> Or if you open the graphics parent module it would open all the child
>> projects as well. And opening the top level openjavafx POM, would open all
>> the child projects and descendants for everything. (As an aside, with Maven
>> I never check in my intellij project files, and tend to get my developers
>> to open the POM file directly - the POM file becomes the source of truth
>> and is IDE agnostic).
>>
>> Obviously this would all make it massively easier for contributors, since
>> we often would be playing only in one module (like controls), so the build
>> would be simple and quick, drawing bits we aren't messing with straight out
>> of your repo, and we never even need to open these other bits in our IDE.
>> This is extra useful given all the nasty native stuff that JFX has - as a
>> contributor I don't want to have to build 'glass' just to add a ComboBox
>> fix, etc.
>>
>> If you have really good clean modularisation, then versioning gets easier.
>>
>> I don't really know how your internal build works, how often it happens,
>> etc, and how you structure milestones. I've heard Agile mentioned but I'd
>> guess it's psuedo-agile. But to give us something to work with, let's
>> assume a pretty standard Agile practice, maybe something like an automated
>> nightly build that compiles all the source code for all the platforms and
>> runs the unit tests, and reports back any failures. Then at the end of a
>> milestone you have a manually triggered build that does all of the same
>> again, tags it in SVN, and your testing team then runs their full set of
>> integration/regression tests on this. Maybe the reality is a bit different
>> for you guys, but hopefully its close enough you can extrapolate.
>>
>> My strategy here would likely be to have the automated nightly builds
>> deploy a SNAPSHOT release (say 8.0.1.SNAPSHOT) which is always the latest
>> and greatest of 8.0.1. I would then also have these automated builds deploy
>> the exact same code as a versioned build - probably something like
>> 8.0.1-2013-07-29 where the last bit is the date it was deployed. Note that
>> when I say 'deploy' here, I mean into your own self-hosted Maven repo
>> (Artifactory, or whatever you want). You wouldn't deploy nightly builds
>> into Maven Central.
>>
>> By default, the checked-in code would reference the SNAPSHOT version, so
>> in every POM file that referenced another module (e.g. controls would have
>> a dependency onto base), it would use 8.0.1.SNAPSHOT as the version (or
>> more likely a variable like ${project.version}). Anyone who checks out and
>> builds would automatically be building against the latest, which is what
>> you would expect.
>>
>> Ideally everyone should be constantly updating their code and making sure
>> it compiles against the latest always. You mention two week old code - this
>> isn't exactly something you'd expect to see in an Agile process. Agile
>> tasks are typically 4 to 16 hours and the conclusion of each should involve
>> a check-in and update to the latest and then be included in the nightly
>> build and unit tested against, etc. If a task is bigger than that, you're
>> probably drifting from Agile best practices (and typically would break your
>> task down into smaller chunks). Your nightly builds should always run off
>> head and compile so everyone should be working towards that for their
>> milestone (and milestones should be 1 to 4 weeks in a good agile process,
>> with all code checked in by the end of the milestone, then tagged and
>> released for user feedback).
>>
>> But assuming you had reasons for these out of date or very long tasks,
>> then to pin your code to a particular version, there are a couple of
>> options.
>>
>> The first option is the simplest: just have the entire openjfx project
>> open in your IDE. When you have a module open and included in the IDE build
>> (in IntelliJ at least, I don't know what they others do) then your copy of
>> the code is always used. Even if a newer snapshot is deployed, it's not
>> used by IntelliJ (I'm pretty sure of this, but it could use double
>> checking). So if you have all the modules open (i.e. you opened the top
>> level openjfx POM, which is what I assume most of you do) then you control
>> what versions you are using, simply by controlling the source control
>> updates. If you don't want a newer version of 'base' don't update it from
>> SVN or update to a specific SVN revision.
>>
>> The second option is more explicit. Just change the dependencies in your
>> POMs between modules to reference a specific version. So if you are on the
>> controls team and you need to hook into a specific version of base, you
>> just edit the controls POM and where it references base as a dependency put
>> in the version number for the day you want to pin to, e.g. version
>> 8.0.1-2013-07-29. You are then guaranteed to be fixed against that version
>> and never have to worry about a change being forced on you.
>>
>> The versioned nightly builds are purely to allow for the second option. If
>> you didn't need the second option, you could just use snapshots. If you do
>> want these versioned nightly builds then they are going to build up over
>> time and given the size of JFX this might get to be a problem after a
>> while. Personally I would keep these daily versions around only for a
>> milestone. At the end of a milestone I would do a formal milestone release,
>> like 8.0.1-m23 (where m23 is the milestone number) and deploy that version.
>> People really shouldn't be sticking to older code for long (it's not
>> healthy) so I would delete all the old daily builds (you can easily do this
>> in your own hosted repository, unlike Maven Central). Anyone who really
>> needs to stick to that milestone would at least have to upgrade to the
>> final version of that milestone. If this is too harsh for your needs, you
>> could keep the last 3 (or 5 or 20 or 100) milestones worth of nightly
>> builds.
>>
>> I assume you do several internal milestones releases for each actual
>> release to the public. If so, I probably wouldn't release milestones to
>> Maven Central but I'd consider putting each 'release' into there (e.g.
>> ea-b96 is a release). Or maybe you just wait until you get out of 'ea' and
>> do actual formal releases such as 8.0.1. These non-ea releases definitely
>> should be synched to Maven Central - this is the bit where we run into
>> trouble with Oracle policy on self-hosting, but if you had your own private
>> repo where you were releasing as per all of the above, with POMs all setup
>> and source code and javadoc included, it would be a fairly trivial task for
>> us in the community to then deploy your formal releases to Maven Central
>> for you.
>>
>> If any of that is unclear or needs further elaboration, let me know and
>> I'll do my best. It's a lengthy topic and difficult to talk through like
>> this.
>>
>> As usual this is all my experience, my opinions and my info that I have.
>> Take and use anything you want, ignore the rest. And if people out there
>> have better ways of doing things, I'd be just as keen as anyone to know
>> about them.
>>
>>