Dynamic modules and bidirectional interoperation

Fri Oct 21 13:24:38 UTC 2016

Hi Mark,
I see an other way to solve the wiring issue,
use VM anonymous classes (via unsafe.defineAnonymousClass) to generate the glue code (as lambda meta factory does to implement the functional interface at runtime) and method handles as wires (once created a method handle do not do any security check anymore, so the requirement to be in a module with specific parents can be lifted). With that, you don't need to generate the wiring code inside a new module and when a new bundle need to be re-wire, you can mutate the wire (the method handle) by using a mutable callsite.

Said differently, loading/unloading an OSGI bundle is equivalent to doing monkey patching in a dynamic language, so there is no reason to not use the tools we provide for dynamic languages to wire OSGI bundles.

so i vote strongly against that proposal mostly because it's too late to change requirements and because some obvious design options to implement OSGI on top of JPMS have not been tested.

regards,
Rémi

----- Mail original -----
> De: "mark reinhold" <mark.reinhold at oracle.com>
> À: jpms-spec-experts at openjdk.java.net
> Envoyé: Jeudi 6 Octobre 2016 19:07:59
> Objet: Dynamic modules and bidirectional interoperation

> The agreed requirements for this JSR mandate a modest degree of dynamic
> configuration [1].  They do not mandate _dynamic modules_, i.e., the
> fine-grained dynamic configuration or "hot deployment" that is possible
> with other module systems, such as OSGi and JBoss, in which individual
> modules can be installed, updated, and uninstalled at any time.
> 
> The requirements also mandate that it "be possible for another module
> system, such as OSGi, to locate Java modules and resolve them using its
> own resolver, except possibly for core system modules" [2].  They do not
> mandate _bidirectional interoperation_, which in the case of OSGi would
> be the ability to arrange for bundles to refer to JPMS modules and also
> vice versa, i.e., for JPMS modules to refer to bundles.
> 
> As things stand, the agreed dynamic-configuration and interoperation
> requirements appear to be satisfied by the proposed design.  We have
> several open issues, however, among them #NonHierarchicalLayers,
> #MutableConfigurations, and #LazyConfigurationAndInstantiation, arising
> from the desire for dynamic modules expressed by David Lloyd.  More
> recently, Thomas Watson reported on his attempt to achieve bidirectional
> interoperation between OSGi and JPMS [3], in which he devised a way to
> for a layer of JPMS modules to reference a layer of OSGi bundles.  For
> his approach to be practical, however, he needs to be able to discard
> sub-graphs of modules within a layer so that they and their class loaders
> can be reclaimed earlier than the layer itself (#DiscardableModules [4]).
> 
> At this stage we could proceed to adopt dynamic modules and bidirectional
> interoperation as two additional requirements, to be supported directly
> by the module system.  Doing so would, however, lead to a very different
> and more complex design, with correspondingly complex implementations,
> since it would need to provide all of the features mentioned above, and
> quite likely more.  As I've argued elsewhere this would, in my judgement,
> make it very difficult -- if not impossible -- to achieve the stated
> goals of this JSR, most importantly the goal that the resulting module
> system be approachable by Java developers of all skill levels.
> 
> Fortunately, I don't think we need to go that far.
> 
> Rather than move toward a design that supports these requirements
> directly we can instead support them with the present design indirectly,
> by modeling a dynamic module as a sequence of JPMS modules over time.
> 
> This key idea is latent in Watson's OSGi adaptation, in which he models
> OSGi bundle wirings -- rather than bundles themselves -- as JPMS modules.
> When bundles are installed, updated, or uninstalled then he synthesizes
> new JPMS modules for the resultant new wirings and configures them into
> a single new JPMS layer, using the new class loaders created by the OSGi
> framework itself.  That _bundle layer_ refers to the previous bundle
> layer, if any, so that over time a stack of bundle layers builds up in
> which the topmost layer always represents the current bundle wirings.
> 
> JPMS modules can be configured and resolved in additional layers above
> the current OSGi bundle layer.  If a bundle upon which a JPMS module
> depends is updated then the layer that contains that JPMS module is
> discarded and a new layer is instantiated, configured to refer to the
> new topmost bundle layer and (I assume) from the same set of root JPMS
> modules as the original.
> 
> A big problem with this scheme, as Watson points out, is that the JPMS
> modules corresponding to discarded OSGi bundle wirings can be retained
> in memory for a long time, along with their class loaders, class objects,
> and related data structures.  That's because every bundle layer remains
> in the stack of bundle layers, and that stack is not discarded until the
> OSGi framework itself is stopped and discarded (JPMS-ISSUE-007 in [3]).
> 
> Watson suggests solving this problem by allowing sub-graphs of modules
> within a layer to be discarded (#DiscardableModules [4]).  That would,
> however, require layers to mutable, whereas today they are not.  That
> layers are immutable brings a number of advantages, including a simpler
> conceptual model, a stronger degree of reliable configuration, and safer
> implementations.
> 
> Here, then, is an alternative approach:
> 
>  - Map each bundle wiring to a unique JPMS module, configured and
>    instantiated in its own unique bundle-wiring layer.  An OSGi bundle
>    will then be associated not only with a sequence of bundle wirings
>    over time, but also with a sequence of JPMS module descriptors,
>    configurations, modules, and layers over time.  A layer can be
>    reclaimed after no strongly-reachable layers refer to it, so
>    discarded bundle-wiring layers -- and their class loaders -- won't
>    be retained unnecessarily.  There's no need to represent multiple
>    versions of the same bundle within a single layer, since every
>    bundle wiring has its own layer (part of JPMS-ISSUE-006 in [3]).
> 
>  - Extend JPMS with #NonHierarchicalLayers, so that a set of JPMS
>    modules can be resolved against modules in more than one parent
>    layer.  An ordinary JPMS layer can then have multiple bundle-wiring
>    layers and ordinary JPMS layers as parents.
> 
> To illustrate, here's a diagram that elaborates upon the second diagram
> in Watson's note:
> 
>    http://cr.openjdk.java.net/~mr/jigsaw/notes/osgi-on-jpms.png
> 
> The left-hand side shows a layer containing two ordinary JPMS modules,
> jpms.a and jpms.b, resolved against two OSGi bundle-wiring layers, for
> bundle.a and bundle.b.  (The blue boxes represent JPMS modules; the
> orange boxes are the class loaders created by the OSGi framework.)  The
> right-hand side depicts some later time after which bundle.b has been
> updated, so its original JPMS module and layer were discarded and a new
> module and layer were created for it (bundle.b layer 2).  The JPMS module
> layer had the original bundle.b layer as one of its parents so it, too,
> was discarded and a new JPMS layer was created with the second bundle.b
> layer -- along with the original, unchanged bundle.a layer -- as its
> parents.
> 
> This implementation of OSGi bundles in terms of JPMS modules leverages
> the fact that OSGi already associates a bundle with a sequence of bundle
> wirings over time, and each new bundle wiring has its own unique class
> loader.  All we are doing here is wrapping each of these class loaders in
> its own module and layer, so that ordinary JPMS modules can be configured
> to refer to them.  There could be many such layers but these layers are
> very lightweight, since their sole function is to associate a single
> module with an existing class loader for the purpose of access control.
> This approach should generalize to other module systems that support
> dynamic modules in a similar fashion, i.e., by creating a new class
> loader each time a dynamic module is updated.
> 
> There are, no doubt, further details to work out, such as exactly how to
> synthesize the module descriptors for bundle wirings so that they can
> sensibly be referenced by ordinary JPMS modules.  Cyclic relationships
> amongst OSGi bundles and split packages across bundles can't be modeled
> directly in module descriptors, but I suspect they can be accommodated by
> inserting the necessary additional readability edges after the relevant
> layers are instantiated.  Overall this does, however, appear to be a
> promising approach to supporting dynamic modules and bidirectional
> interoperation without adding undue complexity to the platform module
> system itself.
> 
> To enable this with the present design we do need #NonHierarchicalLayers;
> that's a significant change, but I think it's feasible and will explore
> it further.  In order to model the cyclic graphs and split packages that
> are possible in other module systems it'd be convenient, but not strictly
> necessary, to have #ReadabilityAddedByLayerCreator [5].  We have no need
> for #MutableConfigurations, #LazyConfigurationAndInstantiation, or
> #DiscardableModules, which I think is a big win for simplicity.
> 
> Comments?
> 
> - Mark
> 
> 
> [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#dynamic-configuration
> [2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation
> [3]
> http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-August/000062.html,
>    or http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html
> [4] http://openjdk.java.net/projects/jigsaw/spec/issues/#DiscardableModules
> [5]
> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReadabilityAddedByLayerCreator