The critical missing pieces and a path forward

Mon May 8 14:57:36 UTC 2017

2017/5/5 11:15:47 -0700, david.lloyd at redhat.com:
> Fellow experts,
> 
> We and other EC members and community members have, on several 
> occasions, communicated the various deficiencies we see in the 
> specification.  This is an update to make sure that it is very clear 
> which few technical, objective criteria are being missed by the current 
> specification that actually pushes it over the line to unacceptability 
> to Red Hat in its current form.  I can't speak for other EG or EC 
> members, but I believe this accurately summarizes our position.

Thank you for the summary.  We've discussed all of these issues before.
Since then I've seen no new information that persuades me to change my
views on them, but I'll summarize those views here for the sake of the
public record.

First, however, a broader point.

Part of our collective role here is to act as responsible stewards of
the Java Platform.  If we are to take that at all seriously then we must
think as carefully as we can about every change or addition that we make.
Every change to existing functionality carries compatibility risk.  Every
addition places hard constraints upon how the platform can evolve in the
future.  That a change or addition appears useful to some is just the
start of what must be a thorough analysis of its wider and long-term
consequences.  That a change or addition is "quite easy" to implement
is completely irrelevant.

History bears witness to this.  In the early years of Java we sometimes
chose to expose low-level functionality, or violate fundamental language
invariants, or leave APIs open for extension without having designed them
for that purpose.  In each case it was convenient and expedient and, yes,
"quite easy", but twenty years later we're still paying for many of those
decisions.

To cite just one example, consider the fact that `final` does not really
mean "final", which I've mentioned before [1].  The core reflection API
introduced in Java 1.1 brought the ability to change the value of `final`
fields.  This seemed useful, and implementing it was "quite easy" at the
time, and it did not have any obvious impact on the rest of the system.
In the long run, however, it has greatly complicated if not prevented JIT
compiler optimizations based upon or enabled by constant folding -- which
is an awful lot of them.

In the context of a library or an application the consequences of any
particular design choice are often apparent, and it's often possible to
correct a bad choice over time.  We're working here, however, on the Java
Platform itself.  Millions of developers and users and customers depend
upon it, every day.  In that context the consequences of any particular
design choice are often difficult to predict -- and bad choices are
sometimes impossible to correct.

If responsible stewardship means anything then it means that we must
reject any proposal that merely appears useful to some and is "quite
easy" to implement.

To act otherwise would be the height of irresponsibility.

Now, on to your criteria.

> The first criterion is to allow cycles to exist among modules at run 
> time.  Our experience with deploying graphs which include third-party 
> modules leads me to believe that this is going to be real problem that 
> people encounter, which will be surprising and (in some cases) very 
> difficult to fix or work around.

>From previous conversations my understanding is that your experience is
based upon deploying graphs of existing unchanged non-modular JAR files,
with all of their messy flaws, in a Java EE application server.  From a
purely technical perspective, it's impressive that you can do that.  If
we were aiming in this JSR only to describe and standardize that practice
then I would agree that cycles should be allowed.

A primary goal of this JSR is, however, to create a module system that's
approachable by all developers -- not just developers of application
servers -- for use in the construction, maintenance, and deployment of
libraries and large applications.  This demands that we think more
broadly about how to help all developers write programs "in the large".
One result of that thinking is that this module system imposes some
constraints that are intended to make working systems easier to reason
about, and to guide developers toward better practices.  Adapting to
these constraints will sometimes require changes to existing code, but
we've assumed from the start that modularization is an active process
undertaken by the maintainer of a component rather than something done
after-the-fact by someone else who cannot modify the component.

Forbidding cyclic module relationships is one of these new constraints.
That such relationships are generally a bad idea is not just my personal
view; see, e.g., Kirk Knoernschild's compelling arguments against them
in his fine book, _Java Application Architecture_ [2], specifically in
Â§4.4, "Cyclic Dependencies -- The Death Knell".

As I wrote previously [3], if real experience with this module system as
used directly by developers reveals that cycles should be allowed then in
a future release we can revise the specification to allow them.  If we do
so now, however, then we can never go back on that choice.

(For those not familiar with the details of the design: Cycles are not
 allowed when modules are resolved but it is possible to create them
 post-resolution via the API, if needed [4][5].  Cycles are also set up
 amongst all automatic modules, after resolution, to make it easier to
 treat existing JAR files as modules without change [6].)

>                                   On the other hand, it should be quite 
> easy to allow this within the module resolver code, especially given the 
> self-contained and all-at-once nature of its algorithm.

The ease with which a change can be implemented has no relevance at all
to whether it is the right change for the Java Platform for all time.
(See above.)

> The second criterion is to re-evaluate (piecewise if necessary) the 
> (still very small) module primitives patch.  This will be a huge boon to 
> container developers and framework authors (yes, this includes us, as 
> well as others).  This is a very small effort even if the entire patch 
> is taken

The ease with which a change can be implemented has no relevance at all
to whether it is the right change for the Java Platform for all time.
(See above.)

>          (which is something that, as I've said before, we're open to 
> discussion on, and there is much we can discuss and compromise on, on a 
> point-by-point basis, if only such discussion could be allowed).  This 
> bridge may open the path to allow the *entire* Java ecosystem to begin 
> to converge on Jigsaw in ways beyond the class path, and indeed might 
> mean the difference between unification and fragmentation.  Mark himself 
> has admitted that the specification isn't perfect, which (in my opinion) 
> also strongly argues in favor of the sensible strategy of providing a 
> hook for extensibility.  This concession could provide relief for _many_ 
> of the other problems that we and others have identified.

As I wrote earlier [7], the primitives that you propose to expose were
not designed for use outside of the reference implementation itself.  To
expose them would constrain the future evolution of this module system,
and the rest of the platform, with a commitment to support them for the
long term.  They were not designed for that purpose.

To design a proper set of such primitives, i.e., a "meta" module system,
is not a goal of this JSR.  If that had been a goal then we would have
started by studying the internals of many existing module systems in
order to identify a common set of primitives to which we would be willing
to commit for the long term.  That's not what we've done, however, and
without having done that I can't be confident that what we have today is
the right set of primitives, or even a good one.  To commit to this set
of primitives now would be irresponsible.

> The third criterion is to provide package namespace isolation among 
> module path modules.  I maintain that the easiest way to do this is to 
> isolate each module path module to its own class loader, which also 
> should greatly ease the transition for existing Java code that uses the 
> existing ClassLoader API.  It is clear to myself and others that package 
> namespace isolation is a fundamental expectation of a module system; the 
> few existing widely-deployed module runtimes have this behavior.  Our 
> own experience tells me that this is going to be a real problem for many 
> nontrivial applications.  Not having this type of isolation dramatically 
> undermines the value of the system.

I understand the concern here, but the impact of not isolating package
namespaces by default is ameliorated by the fact that most developers use
the broadly-accepted reverse-DNS convention for package names.  If an
application works properly on the class path today then it likely does
not have conflicting packages to start with, since conflicting packages
on the class path so often lead to trouble.

This module system loads application modules into a single class loader
by default so that developers have one less behavioral difference to
worry about as they upgrade their components to explicit modules.  An
early prototype did load every module into its own loader by default.
With that we found that a fair bit of existing code -- not just code in
the JDK itself -- is written assuming that the grandparent of its loader
is the boot loader, that its loader (or perhaps its loader's parent) is
the thread-context loader, that code in nearby modules is in the same
loader, and so on.  In a future release we can explore loading every
application module into its own loader, and maybe even make that the
default behavior over time, but for now a more conservative approach is
preferable.

(If do you need to load a set of modules with each module in its own
 class loader then you can arrange for that in just a few lines of code
 via the API [8].)

- Mark

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-April/011983.html
[2] http://www.kirkk.com/modularity/
[3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-March/000646.html
[4] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/Module.html#addReads-java.lang.Module-
[5] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/ModuleLayer.Controller.html#addReads-java.lang.Module-java.lang.Module-
[6] http://openjdk.java.net/projects/jigsaw/spec/sotms/#automatic-modules
[7] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-February/000585.html
[8] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/ModuleLayer.html#defineModulesWithManyLoaders-java.lang.module.Configuration-java.lang.ClassLoader-