Transitive deps was: Jigsaw prototype, take 2

[David M. Lloyd]

On 09/05/2013 05:39 AM, Neil Bartlett wrote:
> On Thu, Sep 5, 2013 at 8:26 AM, Jaroslav Tulach
> <jaroslav.tulach at oracle.com> wrote:
>> Dne St 4. září 2013 19:50:34, David M. Lloyd napsal(a):
>>> * Transitive-by-default causes problems in mid to large projects due to
>>> extensive conflicts [can't find the discussion...];
>>
>> We run into this all the time. Just recently I was trying to use maven-antrun-
>> plugin which depends on (and exposes) Ant 1.7.1 while using features of Ant
>> 1.8. Conflicts everywhere. Resolution fragile. But it seems to run at the end.
>>
>>> fix is to use and
>>> verify exclusions, specify "provided" scope,
>>
>> Provided scope is good for "compile only dependency" - e.g. one that is not
>> used during runtime. I use it for APIs with annotation processors that just
>> generate something and are no longer needed during runtime.
>>
>> However I've noticed that grizzly used the "provided" scope for optional
>> runtime dependency. E.g. I could use grizzly-http-server, and add in optional
>> support for websockets - but it was a nightmare to find out all the JARs that
>> were needed (like javax.servlet-3.0 API) and I got a lot for
>> NoClassDefFoundErrors before I collected them all. I double this is a practice
>> to follow.
>>
>>> and use
>>> maven-enforcer-plugin [5] to ban transitive dependencies
>>
>> That might work. Explicitly specify what you compile against is a good idea.
>> NetBeans Platform (Ant based) build system is using it for a decade (after
>> switching from the previously transitive mode) and there are no problems with
>> it as far as I can tell.
>
> I continue to believe that the root of this problem is that the Maven
> dependency model cannot work for runtime dependencies. "Whole module"
> dependencies such as used in Maven, in previous Jigsaw/JSR277
> prototypes, and in OSGi's discouraged Require-Bundle instruction
> always lead to excessive fanout and version clashes.

We use whole module dependencies all the time in JBoss Modules without 
such problems; however we don't always transitively pull in all imports 
either, which might be a critical difference, and we also provide 
mechanisms to limit what is exported from the source module as well. 
All the OSGi arguments I can find seem to stem from only having two 
levels of granularity: packages, or *everything*.  Really I guess it's 
probably more fair to say that our "whole module" dependencies are 
really a midway point between OSGi's per-package and whole-bundle 
dependencies; they really mean "the exported subgroup of all of the 
module's packages".

In any case this has worked better than package granularity for us 
because it allows APIs to grow new packages without having to repackage 
every dependent (something that happens a lot with rapidly developing 
projects like ours), while still allowing us to hide packages and 
transitive imports that should not be imported into the target.

This is one of those fine details that would make for an interesting 
discussion on its own, but doesn't really relate much to the original 
topic :)

But the #1 reason Maven dependencies don't work in practice is simply 
that they're designed for a flat class path.  More than one of my 
colleagues have naively tried to come up with a mechanism that 
automatically just maps Maven dependencies on to JBoss Modules, using 
the exact literal dependency resolution from Aether or whatever 
mechanism they try (both with multiple versions and flattened versions), 
and frequently they have run into NCDFE/CNFE and other linkage errors, 
CCE, etc.  The dependencies in the existing repository are not adequate 
(let alone deterministic [version ranges!]), and the dependency 
specification language is not expressive enough to accommodate even the 
most common use cases we have found (e.g. establishing what packages are 
non-public).  Being transitive by default is a big part of the reason 
why this space is "poisoned", in my opinion, but also the side-effects 
of a flat class path (i.e. forgotten dependencies magically still work 
through luck) will always prevent this database from working in an 
isolated module system until/unless Maven itself moves to an isolated 
system.

>>> * range dependencies can cause resolution
>>> to be NP-complete; best practice is to use ranges in a more restricted
>>> manner [6]
>>
>> Thanks for the quote. Btw. the article's final suggestion is somewhat similar
>> to what the maven-enforcer-plugin does. The write up suggest to record the
>> exact versions of libraries we compile against - either specified directly as a
>> compile dependency, or inferred by the compiler for the (hidden) transitive
>> ones.
>
> Although David quoted you in the context of OSGi resolution, your
> article is not really considered OSGi best practice. In fact we do use
> wide ranges according to the rules of Semantic Versioning[1]. While
> NP-complete resolutions can technically occur, the problem rarely
> arises in realistic scenarios -- if they do ever occur then the
> solution is to narrow the search space and/or pre-resolve and persist
> the resolution state.

In any case though, the way OSGi uses version constraints to dynamically 
wire up dependencies (and verify compatibility) works in a container 
(where you don't actually know what's going to come in and from where) 
but in a static environment like Jigsaw and the default JBoss Modules 
loader, you want all those questions to be answered up front (much like 
in an OS distribution), so while version ranges can be useful to help 
ensure correctness in your original module repository, I think making 
them part of the runtime resolver is a mistake.

This is where the OSGi people talk how OSGi can pre-cache resolution 
results, and yes I acknowledge this is a solution, but only if you 
actually *do* it. :-)  Using this information to build the static module 
repository in the first place essentially gives the same result though.

But in any case, for a Java SE module system, I still maintain that 
modules should always load in constant time (algorithmically speaking), 
and that means no version ranges at run time, period (other than perhaps 
for run time verification purposes, if the integrity of the original 
repository is in doubt for some reason; this might be a core assumption 
though if we're talking about using a Maven repository as a source for 
modules).

> However, I really feel there is no point in rehashing all these
> arguments yet again, since we don't have a clue about the requirements
> of the rebooted Jigsaw project other than Mark's vague implication
> that it will be "simpler". Oracle needs to clarify the purpose and
> goals of the project before any progress can be made.

Indeed, once again we don't know what worked and what didn't work and 
why, and what Oracle is *really* after.  Instead we are expected to be 
lulled by the disclaimer that "this is just another prototype [that you 
know nothing about]".

-- 
- DML