Alternative module system design proposal

[David M. Lloyd]

This email describes an alternative module system design proposal (I'll 
call it the AMSD for brevity) and serves as a partial rebuttal to, and 
possible alternative for, the state of the module system doc that Mark 
just posted (I'll call this the MR approach for simplicity).  Included 
is a link to an example partial implementation (this implementation is 
not intended (technically or legally) to be a contribution at this time, 
but merely an example).

Principles
----------
The key principle of the alternative design is the reduction of 
abstraction.  While MR's proposal (broadly speaking) includes adding 
modules as a new layer (with corresponding reflective changes), changes 
to the language specification, binary descriptors, and a strong 
hierarchy-based orientation, the AMSD equates modules to class loaders 
in a stronger way.  No new layers of abstraction are introduced (though 
class loaders are slightly enhanced), the language specification is not 
changed, and descriptor format remains open.  Modules and their 
configurations (I call them "module loaders") can exist in arbitrary 
graph relationships and are not bound to a strict hierarchy.

A module in the AMSD is a specialization of a class loader, exploiting 
the high degree of commonality between modules (as specified) and class 
loaders as they exist today.

A 'module loader' fulfills the specification role of "configuration". 
Behaviorally, it is much like a class loader, but for modules and not 
classes.

Module loaders can be aggregated.  This is of specific importance for 
bootstrapping; when starting the JVM, only the 'java.base' module would 
initially be available.  A second module loader is instantiated to 
provide access to the remainder of the JDK; to start the user 
application and load its modules, a third module loader is used.  This 
layering makes bootstrap run smoothly, and also encapsulates the 
platform from the application's modules.  See [1] for a visual example.

No additional concepts are introduced into the JVM (beyond what is 
already in jdk9/jdk9 at this time, specifically JRT support), though 
some concepts are enhanced in the JDK.  The implementation would be 
almost 100% pure Java with no (or only very minimal) JVM changes 
necessary to support the run time, with the exception of tooling and the 
java command itself, which would be enhanced to specify modules as 
previously discussed.

Compatibility
-------------
While MR's design hinges upon exactly preserving the existing class 
loader structure, the AMSD seeks only to emulate the certain behaviors 
that were given to justify this preservation.  The primary compatibility 
factor was described to be the assumption by certain internal classes 
that the class loader of certain other internal classes would be 'null'. 
  There exists a provision within the bootstrap module system code to 
allow internal classes to advertise a class loader of 'null', even if 
that class was actually loaded by a module.  A secondary 
getModuleClassLoader() method is specified to obtain the actual module 
that defines the given class.

This flavor of design has been shown to be 100% compatible with Java EE 
(though not all *implementations* of Java EE are necessarily 100% 
compatible with this design, without replacing their class loader 
infrastructure, nor would they be compatible with the MR design) as well 
as with OSGi.

More existing behaviors are preserved by this approach, including being 
able to read .class files as resources (I find it telling that the very 
first public post about testing the MR implementation already tripped 
over this).

Implementation
--------------
A partial example implementation exists at [2] (list of individual 
commits at [3]).  It is (obviously) not complete, but I believe that it 
does contain all of the groundwork necessary to implement all of the 
requirements given in the specification.  Some requirements are not yet 
met as of this date.  I have a detailed outline of this information that 
I can share later if there is interest in this implementation alternative.

Part of this implementation entails adding a formal Resource concept, 
which ties an arbitrary resource to its initiating class loader.  This 
is essential to providing a service loading implementation with adequate 
encapsulation rules.

The crux of the success of this implementation revolves around modifying 
the bootstrap class loading process to only load classes from the 
java.base section of the JRT.  This is not yet done.

The linkage algorithm is based heavily on that of JBoss Modules.  Each 
module class loader contains an index of all packages visible to it; a 
module's class loader can, in constant time, determine whether a class 
or resource is loadable from that class loader, and if so, where that 
class or resource comes from.

There is no prescribed descriptor format.  The module definition API is 
presently based on a builder pattern; any descriptor methodology can 
feed into it directly.  Supporting textual descriptors is a primary 
motivator for this; no such format is yet established in the AMSD 
example code.  JBoss Modules used XML (for reasons which are probably 
not applicable to the JDK); it may be better, from a usability 
perspective, to use a specific textual language that can be parsed directly.

There is no bootstrap support at present.  This precludes loading or 
running modules without external support code; it also precludes things 
like agents, and the ability to register other early services like 
loggers and security managers from modules.

Objective
---------
The goal of this implementation is to spur discussion on certain topics:

* Descriptor encoding and module format
* Resource handling
* Security
* Implementation complexity
* Forwards compatibility
* Programmatic API capabilities

[1] 
https://www.lucidchart.com/publicSegments/view/e5fbad2c-6c3f-402d-93f8-5afad4b90132/image.png
[2] https://github.com/dmlloyd/openjdk-modules
[3] 
https://github.com/dmlloyd/openjdk-modules/commits/modules?author=dmlloyd
-- 
- DML