Why package deps won't work (Was: Re: Converting plain JARs to Java modules)

[David M. Lloyd]

The explanation is quite simple, really - each point can be pretty much 
wiped out by a heavy dose of reality.

1. "M2P is leverages the Java type system unlike m2m that must introduce 
new namespaces outside the Java type system." - this is just fluffy 
buzzwordology.  Considering packages part of the Java type system is a 
pretty liberal interpretation of the term "type system" as they're 
basically just arbitrary name spaces.  That said, there's nothing 
particularly "new" about namespacing modules.  JARs have names. 
Projects have names.  Maven uses its own namespaces for artifacts.  Even 
the primitive JDK extension mechanism uses symbolic names.

Even a simple convention of the parent-most package name for the name of 
a module is very simple to grasp and is in fact the solution we've used 
quite effectively thus far.  Thus implying that module names are some 
new alien concept is really not valid.

2. "M2P can be used to break the transitive dependency chain, m2m 
suffers of excessive coupling." - How about some facts to back this up? 
  I've found "m2m" coupling to be just right.  In JBoss Modules, we do 
not export transitive dependencies by default.  This results in a very 
simple and clean dependency graph between modules.  Using package 
dependencies results in a *far more complex* dependency graph, because 
you need edges for every package even though *most of the time you want 
the whole module anyway*.  Again, real world here.

If we're just going to throw dogma around, I'll put it the other way: 
m2p is a design error by the OSGi spec designers which has since been 
embraced as a religion.  It offers no significant benefit, other than a 
couple of edge cases which are frankly just as well handled by m2m 
simply by adding package filters.  Which, by the way, I haven't seen the 
need to do yet in our 200+ module environment, but which we do have the 
capability to do.

M2P is a solution just itching for a problem.  But you're going to have 
a tough time convincing me that users *want* to have to use special 
tooling because they want to depend on a module which has too many 
packages to list out by hand.  And before you cry "wildcards", be sure 
to consider that some modules use package names which are subordinate to 
other modules' packages, which is a perfectly normal and allowable 
scenario.  Using wildcards for package matching could cause amazing 
levels of havoc.

Using package dependencies means you either must have a master package 
index for linking, or you need a resolver which has to have analyzed 
every module you ever plan to load.  Otherwise, O(1) loading of modules 
is impossible, which is absolutely 100% a deal-breaker for JDK modules 
which must be incrementally installable.  And it forbids having packages 
with the same name in more than one JAR without bringing run-time 
versioning into the fold, which is a terrible, terrible can of worms.

Finally it should be perfectly clear to anyone who has read the original 
requirements document that nothing in this module system should prevent 
OSGi from functioning as it is, so there is absolutely no reason to 
assume that any OSGi implementation is so threatened - especially if m2p 
linking is as superior as has been expressed.  Our module system (which 
is conceptually similar to Jigsaw in many regards) in fact does support 
our OSGi implementation quite effectively without itself implementing 
OSGi's package-to-package resolution (which like I said throws O(1) out 
the window).

On 11/14/2011 01:49 AM, Glyn Normington wrote:
> I look forward to David's elaboration of why he thinks "using packages as a dependency unit is a terrible idea" to balance Peter's clear explanation of the benefits of m2p.
>
> Meanwhile, it's worth noting that, according to the requirements document, Jigsaw is aimed at platform modularisation and the platform being modularised has some non-optimal division of types across packages (see the package subsets requirement) which favour m2m dependencies. (Note that Apache Harmony was developed with modularity in mind and was able to exploit m2p, so platform modularisation per se needn't be limited to m2m.)
>
> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of legacy code modularisation and less applicable to new module development and modularisation of existing code whose division into packages suits m2p. IIRC this was the original positioning of Jigsaw: for use primarily within the OpenJDK codebase and only exposed for application use because it was too inconvenient to hide it.
>
> Regards,
> Glyn
>
> On 12 Nov 2011, at 11:59, Peter Kriens wrote:
>
>> Neither my wrath, nor the fact that I rarely if ever get angry is relevant in this discussion ... This is a technical argument that are solvable by technical people that share the same goals. I prefer package dependencies because they address the excessive type coupling problem in object oriented systems, not because they're part of OSGi. Let me argue my case.
>>
>> Module-to-package dependencies (m2p) are preferable over module-to-module dependencies (m2m) for many reasons but these are the most important reasons:
>>
>> M2P is leverages the Java type system unlike m2m that must introduce new namespaces outside the Java type system.
>> M2P can be used to break the transitive dependency chain, m2m suffers of excessive coupling
>>
>> Since the first bullet's benefit should be clear I only argue the more complex second bullet.
>>
>> A module is in many regards like a class. A class encapsulates members, depends on other members/classes, and makes a few members accessible outside the class. A module has a similar structure but then with types/packages as members.
>>
>> After the initial success of Object Oriented Programming (OO) it was quickly learned that reuse did not take place at the expected scale due to excessive type coupling. The problem was that a class aggregated many dependencies to simplify its implementation but these dependencies were unrelated to the contract it implemented. Since class dependencies are transitive most applications disappointingly became an almost fully connected graph.
>>
>> Java's great innovation was the interface because it broke both the transitivity and aggregation of dependencies. A class could now express its dependency (use or implement) on a contract (the interface) and was therefore fully type decoupled from the opposite site.
>>
>> An interface can act as a contract because it names the signature of a set of methods so that the compiler can verify the client and the implementer.
>>
>> Since a module has a very similar structure to a class it suffers from exactly the same transitive aggregation of dependencies. This is not a theory, look at the experiences with Maven (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) Again, this is not that maven is bad or developers are stupid, it is the same underlying force that finally resulted in the Java interface.
>>
>> The parallel for the class' interface for modules is a named set of interfaces. This concept already exists in Java: a package. Looking at almost all JSRs it is clear that our industry already uses packages as "interfaces" to provider implementations.
>>
>> Therefore, just like a class should not depend on other implementation types, a module should preferably not depend on other modules. A module should instead depend on contracts. Since modules will be used to provide components from different sources managed with different life cycles the excessive type coupling caused by m2m is even more damaging than in c2c. Proper use of m2p creates significantly less type coupled systems than m2m, the benefits should be obvious.
>>
>> Since there are use cases for m2m (non-type safe languages for example) I do believe that Jigsaw should still support m2m. However, it would be greatly beneficial to our industry if we could take advantage of the lessons learned with the Java interface and realize how surprisingly important the Java package actually is in our eco system.
>>
>> Kind regards,
>>
>> 	Peter Kriens
>>
>>
>>
>>
>>
>>
>>
>>
>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote:
>>
>>> I'll just state now that using packages as a dependency unit is a terrible idea, and not some architectural revelation.  That way, Peter's wrath will be largely directed at me. :-)
>>>
>>> On 11/09/2011 08:02 AM, Peter Kriens wrote:
>>>> I agree that tools are needed but we must be careful to not expect tools to stopgap an architectural issue. I think it is important to first do good architectural design leveraging existing tools (e.g. the Java type system) before you try to add new tools. It is such a pity (but all to common) that a design allows for classes of errors that would be impossible with a slightly different design.
>>>>
>>>> Kind regards,
>>>>
>>>> 	Peter Kriens
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote:
>>>>
>>>>> On 09/11/2011 13:04, Peter Kriens wrote:
>>>>>> The issue is that maven problems are not caused because maven is bad or that pom authors are stupid. The reason is that the module-to-module dependency architecture in maven (and Jigsaw) is error prone ...
>>>>> This thread started out with someone asking about adding module declarations to existing JAR files, and in that context, I agree it can be error prone without good tools. I think things should be a lot better when modules are compiled.
>>>>>
>>>>> -Alan.
>>>>>
>>>>
>>>
>>>
>>> --
>>> - DML
>>
>


-- 
- DML