Dynamic modules and bidirectional interoperation

Thu Oct 6 17:07:59 UTC 2016

The agreed requirements for this JSR mandate a modest degree of dynamic
configuration [1].  They do not mandate _dynamic modules_, i.e., the
fine-grained dynamic configuration or "hot deployment" that is possible
with other module systems, such as OSGi and JBoss, in which individual
modules can be installed, updated, and uninstalled at any time.

The requirements also mandate that it "be possible for another module
system, such as OSGi, to locate Java modules and resolve them using its
own resolver, except possibly for core system modules" [2].  They do not
mandate _bidirectional interoperation_, which in the case of OSGi would
be the ability to arrange for bundles to refer to JPMS modules and also
vice versa, i.e., for JPMS modules to refer to bundles.

As things stand, the agreed dynamic-configuration and interoperation
requirements appear to be satisfied by the proposed design.  We have
several open issues, however, among them #NonHierarchicalLayers,
#MutableConfigurations, and #LazyConfigurationAndInstantiation, arising
from the desire for dynamic modules expressed by David Lloyd.  More
recently, Thomas Watson reported on his attempt to achieve bidirectional
interoperation between OSGi and JPMS [3], in which he devised a way to
for a layer of JPMS modules to reference a layer of OSGi bundles.  For
his approach to be practical, however, he needs to be able to discard
sub-graphs of modules within a layer so that they and their class loaders
can be reclaimed earlier than the layer itself (#DiscardableModules [4]).

At this stage we could proceed to adopt dynamic modules and bidirectional
interoperation as two additional requirements, to be supported directly
by the module system.  Doing so would, however, lead to a very different
and more complex design, with correspondingly complex implementations,
since it would need to provide all of the features mentioned above, and
quite likely more.  As I've argued elsewhere this would, in my judgement,
make it very difficult -- if not impossible -- to achieve the stated
goals of this JSR, most importantly the goal that the resulting module
system be approachable by Java developers of all skill levels.

Fortunately, I don't think we need to go that far.

Rather than move toward a design that supports these requirements
directly we can instead support them with the present design indirectly,
by modeling a dynamic module as a sequence of JPMS modules over time.

This key idea is latent in Watson's OSGi adaptation, in which he models
OSGi bundle wirings -- rather than bundles themselves -- as JPMS modules.
When bundles are installed, updated, or uninstalled then he synthesizes
new JPMS modules for the resultant new wirings and configures them into
a single new JPMS layer, using the new class loaders created by the OSGi
framework itself.  That _bundle layer_ refers to the previous bundle
layer, if any, so that over time a stack of bundle layers builds up in
which the topmost layer always represents the current bundle wirings.

JPMS modules can be configured and resolved in additional layers above
the current OSGi bundle layer.  If a bundle upon which a JPMS module
depends is updated then the layer that contains that JPMS module is
discarded and a new layer is instantiated, configured to refer to the
new topmost bundle layer and (I assume) from the same set of root JPMS
modules as the original.

A big problem with this scheme, as Watson points out, is that the JPMS
modules corresponding to discarded OSGi bundle wirings can be retained
in memory for a long time, along with their class loaders, class objects,
and related data structures.  That's because every bundle layer remains
in the stack of bundle layers, and that stack is not discarded until the
OSGi framework itself is stopped and discarded (JPMS-ISSUE-007 in [3]).

Watson suggests solving this problem by allowing sub-graphs of modules
within a layer to be discarded (#DiscardableModules [4]).  That would,
however, require layers to mutable, whereas today they are not.  That
layers are immutable brings a number of advantages, including a simpler
conceptual model, a stronger degree of reliable configuration, and safer
implementations.

Here, then, is an alternative approach:

  - Map each bundle wiring to a unique JPMS module, configured and
    instantiated in its own unique bundle-wiring layer.  An OSGi bundle
    will then be associated not only with a sequence of bundle wirings
    over time, but also with a sequence of JPMS module descriptors,
    configurations, modules, and layers over time.  A layer can be
    reclaimed after no strongly-reachable layers refer to it, so
    discarded bundle-wiring layers -- and their class loaders -- won't
    be retained unnecessarily.  There's no need to represent multiple
    versions of the same bundle within a single layer, since every
    bundle wiring has its own layer (part of JPMS-ISSUE-006 in [3]).

  - Extend JPMS with #NonHierarchicalLayers, so that a set of JPMS
    modules can be resolved against modules in more than one parent
    layer.  An ordinary JPMS layer can then have multiple bundle-wiring
    layers and ordinary JPMS layers as parents.

To illustrate, here's a diagram that elaborates upon the second diagram
in Watson's note:

    http://cr.openjdk.java.net/~mr/jigsaw/notes/osgi-on-jpms.png

The left-hand side shows a layer containing two ordinary JPMS modules,
jpms.a and jpms.b, resolved against two OSGi bundle-wiring layers, for
bundle.a and bundle.b.  (The blue boxes represent JPMS modules; the
orange boxes are the class loaders created by the OSGi framework.)  The
right-hand side depicts some later time after which bundle.b has been
updated, so its original JPMS module and layer were discarded and a new
module and layer were created for it (bundle.b layer 2).  The JPMS module
layer had the original bundle.b layer as one of its parents so it, too,
was discarded and a new JPMS layer was created with the second bundle.b
layer -- along with the original, unchanged bundle.a layer -- as its
parents.

This implementation of OSGi bundles in terms of JPMS modules leverages
the fact that OSGi already associates a bundle with a sequence of bundle
wirings over time, and each new bundle wiring has its own unique class
loader.  All we are doing here is wrapping each of these class loaders in
its own module and layer, so that ordinary JPMS modules can be configured
to refer to them.  There could be many such layers but these layers are
very lightweight, since their sole function is to associate a single
module with an existing class loader for the purpose of access control.
This approach should generalize to other module systems that support
dynamic modules in a similar fashion, i.e., by creating a new class
loader each time a dynamic module is updated.

There are, no doubt, further details to work out, such as exactly how to
synthesize the module descriptors for bundle wirings so that they can
sensibly be referenced by ordinary JPMS modules.  Cyclic relationships
amongst OSGi bundles and split packages across bundles can't be modeled
directly in module descriptors, but I suspect they can be accommodated by
inserting the necessary additional readability edges after the relevant
layers are instantiated.  Overall this does, however, appear to be a
promising approach to supporting dynamic modules and bidirectional
interoperation without adding undue complexity to the platform module
system itself.

To enable this with the present design we do need #NonHierarchicalLayers;
that's a significant change, but I think it's feasible and will explore
it further.  In order to model the cyclic graphs and split packages that
are possible in other module systems it'd be convenient, but not strictly
necessary, to have #ReadabilityAddedByLayerCreator [5].  We have no need
for #MutableConfigurations, #LazyConfigurationAndInstantiation, or
#DiscardableModules, which I think is a big win for simplicity.

Comments?

- Mark

[1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#dynamic-configuration
[2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation
[3] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-August/000062.html,
    or http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html
[4] http://openjdk.java.net/projects/jigsaw/spec/issues/#DiscardableModules
[5] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReadabilityAddedByLayerCreator