Mutable modules

[David M. Lloyd]

On 05/20/2016 09:12 AM, Alan Bateman wrote:
> On 18/05/2016 22:47, David M. Lloyd wrote:
>>
>> I mean in *our* current concept of a module, we can add/remove/modify
>> the contents of a module (its "class path") at run time.  It is up to
>> the user to ensure that doing so makes sense.
> I don't think I can relate to the use case. As you probably know then
> ZIP files have historically had their central directory mapped into
> memory. Removing or replacing a file that is memory mapped will likely
> lead to processes accessing the mapped file to crash (SIGBUS usually).
> So if users are really doing such hairy things they would need a lot of
> insight into what is running and whether the file is opened before
> taking this risk.

No, we don't support replacement of JAR files; more like we symbolically 
remove the resources and add overlays.  Internally, we generally explode 
JAR files for various reasons.  It has nothing to do with JAR files, 
more to do with the logical presence and absence of class files and 
resources.

>> Our modules each correspond to their own class loader: so far so good,
>> we can just have one Module per class loader.  Problem is that we
>> support circularity, and also we support dependencies that go across
>> module systems with isolated namespaces (basically, our module loaders
>> are a higher order of the exact same concept of class loaders).
> If there are cyclic relationships between your modules then it will be
> problematic. Do you see much of this? If you've read Alex's JavaOne
> slides then you'll know that some of us like Kirk Knoernschild's book on
> Java Application Architecture and section "4.4 Cyclic Dependencies - the
> Death Knell" where he poses the question "Are Cycles Always Bad?". I
> don't want to say too much on this topic here as it is listed as an open
> issue on the JSR issues list.

We use circular dependencies in both our static module layouts and also 
in our dynamic deployment system.  I don't think we have a clear path 
out if we can't support them; it will certainly be a difficult situation.

>> Our modules support specifications including the content of the module
>> ("resource loaders") and the dependencies of the module. At run time,
>> custom ModuleLoader implementations can change the resource loader
>> list and/or the dependency list at any time, causing the module to be
>> relinked on the spot; the most useful aspect of this is the ability to
>> incrementally deploy applications which may include circular
>> dependencies.
> Aside from cycles then what other use-cases do you have here? I read
> "change the resource loader list" to mean that the set of resources in
> the module changes, which is a bit weird if those resources are class
> files that have already been loaded. Maybe there is dynamic code
> generation with class bytes generated to the file system or into
> somewhere virtual? Or maybe these resources are something else, data
> files? I'm just trying to understand what you mean as we are using
> differently terminology.

Within our module concept, the "class path" is a sort of colloquial term 
which refers to the series of resource loaders used to locate classes 
and resources within each module.  Functionally it's somewhat analogous 
to the URLClassPath concept in the JDK, in the way that resources are 
sought (i.e. by a linear search from start to end of the list).

This is separate from the module dependency list, which, when combined 
with the resource loader list, is used to construct an index by path 
(which is a superset of packages that includes not just classes but also 
resources) which refers to dependencies and/or internal resources.

When the resource list is changed, all future lookups for classes and 
resources will use the new index.  If there are already classes loaded 
from the previous list, and those classes are sufficiently incompatible 
with the new code, obviously this will result in errors; however, this 
is usually not the case when (say) making incremental changes during 
development.  This is generally an edge case, but it is one which we 
presently support.

Changing the dependency list has effects that are somewhat similar. 
Existing loaded classes which have already linked against classes of the 
previous dependency may malfunction when doing this.  But in the hot 
deployment situation, most of the time these changes are additive, so 
the effect is generally to enable previously unlinked code to become 
linkable.  This could happen, for example, when deploying a JAR some 
time after a first JAR was deployed, which resolves some missing 
dependencies in the first JAR.

>> We also support delegating "fallback" class loading decisions to
>> outside suppliers for dynamic class loading behavior (this was done to
>> support a dynamic OSGi environment).  The ongoing integrity of the
>> system is up to the party doing the relinking (the EE deployer or the
>> OSGi resolver); most of the time it can reason about what is "safe"
>> and what might cause system breakage (but still might be useful to do
>> anyway).  These are the features we can't seem to support under
>> Jigsaw, architecturally speaking.
> This sounds like class loader delegation to resolve types that are not
> in the module.

Exactly.  Our OSGi people have told me in the past that OSGi that can't 
function to spec without this ability.

>> Specifically this includes (but is not limited to) changing the
>> package set associated with a JDK module at run time, something that
>> this native code block makes impossible.  Also the ability to
>> dynamically change module dependencies is an essential ingredient to
>> making this work.
> Suppose that module m has package p and p.C has been loaded. Are you
> saying that you can drop package p from the module?

Yes - although normally you would only drop p if you knew specifically 
that p.C *hadn't* been loaded, for obvious reasons.  It's probably more 
common to *add* than to drop.

> As things currently stand in JDK 9 then packages may be added to modules
> at runtime, the main use case is the dynamic proxy to a public interface
> in a non-exported packages. So I can relate to adding packages for code
> gen cases, I'm less sure about a module starting out as an XML API and
> suddenly changing into a JDBC driver. Do you really mean the same module
> instance?

A more apt example might be to update a part of a module which hasn't 
yet been loaded.

>> In my view, architecturally speaking, most of the constraints imposed
>> by the core module framework should be layer policy.  If the system's
>> core module layer wants to maintain strict, static integrity, name
>> constraints, version syntax and semantics, etc., that's fine, but why
>> should all modules everywhere be forced to the same constraints?
> Using module names as an example, then it should be possible to develop
> a module that is deployed on the application module path or instantiated
> in a layer of modules that a container creates. The author of the module
> (that chooses the name) isn't going to know in advance how the module is
> deployed. I'm not even sure how such a module could be compiled or how
> anyone could depend on it when the characters or format can vary like
> this. I see there is an issue on the JSR issues list so I don't want to
> say any more on this topic.

The idea of a "module" that I keep referring to is one that is described 
in the JSR requirements: "... named, self-describing program components 
consisting of code and data".  A plain old JAR file meets this 
definition just as readily as the current Jigsaw concept.  Java 8 javac 
can compile things that more than meet the definition of "module".  The 
important and relevant part of a module is the ABI it exposes, not its 
name or even its internal structure (which should be encapsulated in 
numerous senses of the word).  This becomes even more clear when you are 
assembling an environment consisting of hundreds of artifacts from 
hundreds of authors.  We package many, many artifacts whose authors 
never gave a thought to modularity at all.

The responsibility of assigning a name to an ABI and behavior has to lie 
with the environment assembler; it just doesn't make sense in the hands 
of the original author, who is necessarily concerned only with the 
parameters of their problem space, and not with that of any greater 
ecosystem in which their project might find itself.  This is the very 
definition of encapsulation.

For an example of why we may need flexible naming, I can deploy an 
application into a Java EE container called "my-cool-application.ear". 
In this case, I might expect my module to be named 
"my-cool-application.ear" or perhaps "my-cool-application".  I might 
expect nested JARs to correspond to modules named by their relative JAR 
locations (including "/" separators).  Really the only constraint is the 
validity of the name on the filesystem.  Java EE certainly places no 
limitations on this today.

I might name my modules based on Maven group and artifact IDs, which 
have different syntax requirements, or by OSGi bundle name.  You get the 
idea.

The point is that, yes, you are correct: you *don't* know how a module 
is going to be deployed; not before Java 9 and likely not after it 
either.  It doesn't even make any sense to define the module name inside 
the module when the name is going to be 100% dependent on the 
environment in which the module is used.

You can't, on one hand, define a universal namespace and syntax for 
modules and their versions in the JDK or establish hard constraints on 
layer and module graph structure, and on the other hand expect other 
module systems with differing existing constraints to unify on the JDK 
module system.  You're basically cutting these systems off at the knees 
and forcing them to reinvent everything, unless you completely 
coincidentally have a system that already conforms to this structure (if 
so, you are either very fortunate or maybe starting off in a rather 
privileged position).

>> There is no way that existing containers and class loading
>> environments (other than, apparently, WebLogic) can conform to
>> Jigsaw's constraints without losing functionality (and I'm trying hard
>> to find ways to make it work).  This is where most of my raised issues
>> are coming from.
> The module system imposes surprising few constraints.  If you are using
> your own class loaders then the delegation needs to respect module
> readability, something that should not be controversial.

See earlier posts about the controversy of making "public" no longer be 
"public".  The email thread is still unresolved and unanswered.

> However it is possible that you are still at the starting line because
> your have a dependency graph with cycles and/or modules that don't have
> names that can be expressed as a Java identifier, is that right?

Yes, and also isolated module namespaces which nevertheless need to link 
with one another.  Also version syntax and schemes which are not 
compatible with the Jigsaw scheme.  But these issues are all raised in 
the document.

>> All these problems seem surmountable to me, but it becomes
>> substantially more difficult when it is necessary to report all of a
>> module's packages to the module when it is created, since this
>> information is now not easily changed.
> I'm surprised that this is an issue as module membership is critical to
> access control.

Sure, I get that, but it's only critical to *new* access control rules 
that were introduced with Jigsaw.  The classical rules wherein 
public=public that we have relied on this past decade would ease this 
situation substantially, since in this case the only reason module 
information would pass to the JVM would be for diagnostics; since each 
class has a module membership, the JVM theoretically would already have 
access to everything it needs for this purpose.

As long as we're talking access control though... the idea of replacing 
this idea with "friend packages" and using them to selectively expand 
package-private access has never been resolved or even seriously 
discussed as far as I can see.  It's been brought up several times and 
basically ignored.  But this idea would allow not only modules but *all* 
Java code to take advantage of better security by removing public 
qualifiers from things that are not public instead of relying on special 
packages (because package identity is defined by class loader and 
package name, meaning modules are not central to the concept though they 
can easily take advantage of it), which makes far more sense to me at 
least (in particular, within the JDK code itself - all those "shared 
secrets" classes!) and imposes far less risk on the access control model 
by keeping it homogeneous instead of making it bi-layered.  I would 
deeply wish to resurrect this discussion at some point.  Readability and 
exports have been nothing but a problem for users as far as I can see; 
the current security model certainly isn't doing me any favors.

-- 
- DML