Conflicting module versions

[Gregg Wonderly]

> On Feb 7, 2015, at 3:42 PM, Tim Boudreau <niftiness at gmail.com> wrote:
> 
> The practicality of software is that specific versions are what you test with.  Subsequent versions may not be compatible and just because you need to use another software package that needs a newer version should not dictate that rework occur.  Yes, it might be a good thing, but...
> 
> I think it's interesting that, for example NodeJS's library delivery model is that you can and will have as many different versions of a library as you want (since NodeJS's namespacing is the filesystem itself, you can have A/node_modules/libbar depending on B which has a different node_modules/libbar).  Since NodeJS is Javascript, if an object from A's libbar meets an object from B's libbar, they can interoperate if duck-typing allows them to (it also means you find out very late in the game if they don't).
> 
> I bring that up to point out that the whole model of "use the libraries on the system" - i.e. linking against unknown versions of things at runtime - is less important today than it once was.  In a world where deploying usually means spinning up a new OS instance which contains known and known-to-work versions of things, the problems of long-lived machines where libraries will be upgraded in-place many times are less worth solving than they once were.

This is a lot of what I do for server software.   But, that is not the only place that I use Java.  Java could make a giant impact on the desktop, still, if we just focused on making Java more about software that is easy to write and trivial to keep working and less about technology.  The AWT thread model and volatile make it hard for many developers to be successful in multi-threaded environments, which AWT’s threading model makes happen without the developer owning that detail directly.  Putting lots of technology in the developers face, asking them to manage all kinds of issues that could be managed for them, is the complexity that continues to eat away at Java’s usability for simple/desktop applications.

Module versions are another place where compatibility and survivability in an ever changing world needs to just happen.  If I deploy a Java application as a simple jar file to someone, and it is dependent on some other support libraries, what should happen on revisions to those libraries?  Should the developer have to engineer some kind of update process (which practically just can’t happen in most cases because we don’t have a Java app store), or should the application, when run, try to validate itself automatically, to collect bug fixes or security fixes?

For me, module management is the end to end ordeal.  The whole lifecycle issue needs to be taken care of.  For example, look at the .Net one-click-install business.  There, the “publish” action causes the developer to have the opportunity to designate how the software might be updated.  Via a URL, a CD, or a filesystem path.  It would be nice if modules could have such designated source information (maven, gradle and others have already done these types of things for Java, but there is no Java specified mechanism).  

I would like to see java have something along the lines of gradle (maven might work out as well, but is a bit less pluggable) be a part of the JDK such that updates to pieces of an application, designated as bug fixes, could be updated.  This would include lots of things about using secure signatures and the opportunity through this work, for a Java app store to appear on the scene.

One of the things that I still want to see work well and reliably is recursive jar: URL references.  It would be great if I could deploy anything, library, or application, as a single jar, and anyone referencing that jar from another library or application, would cause the class loader used, to be able to resolve anything within my jar, including recursive references to things that I packed inside of it.   The update mechanisms should also be able to replace content, recursively inside of such jars.  This would allow a single jar file to always be used for any kind of deployment. 

> What we need the most is a way to cleanly separate and isolate software.   Netbeans has a class loader hierarchy that tries its best to keep things separated.  I like that.  The Java, conventional hierarchical class loader structure helps prohibit lateral views, which for me, create the largest exposure to hassle.  
> 
> What would be good, is to figure out a way to keep objects from one branch of the tree from ending up in another branch of the tree when that branch is not compatible.
> 
> FWIW, Maven has this via the maven-enforcer-plugin - turn on DependencyConvergence and it will fail the build when multiple versions of something are on the classpath.  That means you have to do something when it happens, but it guarantees it won't happen without you knowing it.

Yes, you can count on Maven doing that, but Java has no such specification.

>  
> I think about it as a set theory problem.  The package names and classes and versions represent a "set".   Discarding the version is the problem in considering where overlap can occur.    There can never be a proper intersection between two versions. 
> 
> If we had tools that could analyze these sets to determine that there was in improper intersection in visibility, that would help make it clear to developers that their code was interacting in a way that incompatibility and class cast exceptions would occur.  
> 
> Then, having a way to unmarshal an old version and remarshal to the new version as a "proxy" or "delegate" through lambdas would be awesome.
> 
> What you're asking for is duck-typing in Java :-)  

In a sense, I want something like duck-typing to work, but I want it to be something that a developer controls and designates the exact details of.  At the lowest level, this is not much different from Serialization in that we take the object, extract the data, hand it to another object and deserialize it into another object.   But, the use of serialization demands, as you illustrate below, that the SerialVersionUid would be the control.  Practically, it would be more about a factory like mechanism that when you hand it one version of an object, it returns a different version.

> Or, really, something more like a classloader that can see both versions of a library, and if the serialVersionUID that would be generated for both is identical, to treat them as interchangeable.  Which is theoretically implementable, but would either require a lot of runtime type information that doesn't exist now, or would impose a substantial performance penalty at runtime (loading any class would involve some search and compare operations).
> 
> Anyway, I fear we're wandering off topic here.  Do you believe that this JSR should implement these things, or just facilitate the possible implementation of them?

One of the largest problems with Java from my perspective, is that things which we know are good and useful, are available in multiple, incompatible packages that cause software to be separated rather than usable more universally.   Things like ant, maven and gradle are mirror images of the same things that have happened on the web with javascript and CSS based technologies.  You have to commit to using something in a way that makes it impossible to use other things.  The divergent evolution caused by Type-A personalities or “fighting across the isle” causes people to lose out on adopting better technologies.   This is mostly people issues and their inability to actually see the good that comes from working together and looking at commonalities.

There is a video on youtube where Simon Peyton Jones designates Haskell as being useless due to it inherently having no output (https://www.youtube.com/watch?v=iSmkqocn0oQ).  In that video, he illustrates though how language designers, looking at other languages for what users like and need there, can help migrate their own language design toward something that works, ultimately, for the good of more people.

I think it would be great for this JSR team to look at drawing such an illustration with the real goal of being “perfect”, and then figure out where to start with ground work that makes maven, gradle, ant and other build systems have an easier time at collecting components, and then help developers manage such components in a way that normalizes dependencies, references and isolation strategies into concrete details that can be continuously refined and enhanced.

If you don’t include the community at large, which means gradle, maven and ant owners/users and you don’t manage to actually get buy-in, then we will have another Jenkins episode and there will be further separation in the community, all for no good.

Gregg Wonderly

> 
> -Tim
>