Comments on the module-file format

[Brian Pontarelli]

I break things down into Build and Runtime. Here are my thoughts about each (sorry for the lengthy reply):

Build
Need hash checking. I prefer in a separate file because then I can download the hash first and cache it and then later download the entire artifact. This saves on download overhead for the MD5. However, including it in the artifact should be fine.
Dependency groups 
compile
run
provided
test-compile
test-run
etc
Version compatibility
http://code.google.com/p/savant-build/wiki/1_5VersionCompatibility
Is 1.0 compatible with 1.1? 
Can we just drop 1.2 in the classpath instead of 1.1? 
Is the project compatible across major versions, minor versions or patch versions? 
Does the compatibility change between versions (1.x is minor compatible but 2.x is patch compatible - this can also happen inside a range where 1.1-1.3 is minor and then 1.4+ is patch)?
Transitive dependencies
This should be stored separate from the artifact so that the entire dependency graph can be built by downloading many small files
You can prune the graph before fetching the actual artifacts, which makes things much faster
Workflow process used to search 1 or more repository for artifacts and how they are cached locally
This should be stored external to the project (VERY IMPORTANT) so that it can change over time and not prevent historical versions from building
Each project should be able to define a custom workflow or use a general one (Savant uses a search method for this based on the UID of the project)
Security here is nice. Savant supports HTTP authentication and HTTPS
I'm not a fan of version ranges or upgrading
This can cause builds to break "magically"
This can cause tests to break "magically"
The version being used is not always known
This can cause two versions to end up in the application if it isn't properly cleaned
Consider a project using the artifact example.com:my-project:my-artifact:[1.0-]
The first build puts version 1.3 in the WEB-INF/lib directory
The second build puts version 1.4 in the WEB_INF/lib directory
WEB-INF/lib now contains two versions of the same artifact and the one that is used is based on how the classload loads jars
Released artifacts can't be updated in place (this is the same as SNAPSHOT versioning)
This breaks builds
Can cause major pain and force developers "rm -rf" their local caches
Can really mess up runtimes
Can lead to multiple versions of the same artifact version
i.e. if you MD5 example-1.1.jar on your 50 developer workstations you might find 10 different versions all with minor code changes that could impact the runtime of the system
Simple and effective release process
My release process does a bunch of stuff including SVN checkout, SVN tagging, SVN branching, running tests, creating JavaDoc, publishing artifacts, etc. 
And it is just one simple command which makes releasing simple and fast
Publishing artifacts to local and remove repositories (via FTP, SCP, SVN, CVS, etc)
Integration builds to support working across projects
http://code.google.com/p/savant-build/wiki/1_5IntegrationBuilds
This prevents situations where the same application has been built on many different developers workstations and contains different versions of SNAPSHOT dependencies
Increases speed by reducing the need to perform full releases of dependencies


Runtime
No auto-upgrades
This can cause the application to "magically" break on subsequent starts without any changes occurring
Multiple versions of the same artifact
This helps prevent the need for jarjar and other similar solutions
Depends on if public APIs return or consume dependencies
Dynamic languages handle this better as you can pass objects between classloaders without errors since all invocations are dynamic. This allows 2 versions of a class to be used and shared across modules
This brings in a whole mess of other concerns though
Same workflow process for fetching artifacts
Ability to pre-fetch artifacts prior to start up of the application
Ability to migrate local caches across environments to avoid pre-fetch
Artifactless startup would be nice
java -module example.com:foo:foo:1.7
Downloads everything on the fly
Runtime updates/upgrades by strict versioning
Ability to specifically upgrade an artifact to a specific version
Runs a version compatibility check
Hot swaps the artifact
This would be really nice to have access to an API during this life-cycle so that you could halt requests, or slow roll requests during an update. Also be nice to have control of when the old version is removed from service

That's a good chunk of what I've been thinking. Obviously my realm has been mostly in the Build area with Savant and that's why I have a lot more insight there. However, I was involved in the runtime versioning and management system at Orbitz and we worked on a number of the runtime aspects as well. 

-bp



On Feb 5, 2010, at 1:18 PM, Mark Reinhold wrote:

>> Date: Fri, 05 Feb 2010 09:33:42 -0700
>> From: Brian Pontarelli <brian at pontarelli.com>
> 
>> ...
>> 
>> Completely understand and agree. I just want to bringing this up again
>> because my knowledge thus far of Jigsaw is that it is based on three
>> things that I feel will make it DOA:
>> 
>> 	- SNAPSHOTs
> 
> I wouldn't say that Jigsaw is "based" on snapshots.  We haven't yet made
> a specific decision as to whether in-place updates should be allowed.
> You've made some interesting arguments for forbidding them.
> 
>> 	- No (or poor) version compatibility
> 
> Not sure what you mean by this -- can you elaborate?
> 
>> 	- Automatic upgrading/downgrading
> 
> If by "automatic upgrade" you mean the ability to say "give me version
> 1.2.3 or later of module 'foo'", then I think it's essential for some use
> cases.  Developers of small client applications, e.g., often prefer to
> test against all (or most) available versions of a library (or platform)
> so as to increase the chances that their applications will run without
> having to download whole new versions of that library (or platform).
> 
>>                                   ... The metric should be to support
>> enterprise applications with 20+ internal artifacts and 20+ external
>> artifacts at a minimum.
> 
> That's certainly one metric, but there are others.  Not every Java
> application is written for enterprise settings.
> 
> - Mark