Artical on JPMS integration with existing module systems

[Thomas Watson]

Below is the plain text format from an article I wrote on using JPMS 
layers with an existing module system (OSGi).  Please excuse the poor 
formatting.  I tried to make it is pretty as I could in plain text to 
allow Mailman to let it through.  I have omitted the diagrams to avoid 
this message getting held up by Mailman also.  If the diagrams are 
important I can post them some other way later:


In my previous blog post I discussed an experiment that creates a JPMS 
Bundle Layer which can represent resolved OSGi bundles as JPMS modules. 
This would allow child JPMS layers to be created that have modules that 
require the OSGi bundles as they would any other module.

In that experiment I took a hybrid approach where the Framework 
implementation and the OSGi bundles themselves did not really live in the 
JPMS world.  Instead I dynamically created a layer on top that attempted 
to represent that OSGi world indirectly within the JPMS world.  Then real 
JPMS modules could be configured to work on top of this facade layer that 
represented the OSGi world.  This can be thought of as taking a top down 
approach to migrating to JPMS.  Unfortunately this approach has a major 
shortcoming because all classes that are loaded in the OSGi bundle layer 
will be associated with an unnamed module.

The fact that the bundle classes are associated with an unnamed module 
caused me to have to do a major hack to grant access to modules 
representing the OSGi bundles.  This hack involved injecting code into the 
jpms modules which could invoke the addReads method in order to grant the 
necessary access to the unnamed module of the bundle class loaders.  This 
does not seem like a real viable solution for running JPMS modules on top 
of and OSGi bundle layer.

I learned much about how the JMPS layer works during that experiment.  The 
hybrid has a major flaw because the delegation graph of class loaders 
involved are not associated with named modules all the way down.  A better 
way would be to do a bottom up approach where each layer involved has 
class loaders which are mapped to one or more named modules.  This way 
when the JPMS layer resolves the modules on top it will automatically 
grant read access as normal from a requiring module to all of its required 
modules it got resolved to.  The following diagram illustrates how the 
layers would look:

[IMAGE REMOVED]

The boot layer contains the JPMS modules which were configured with the 
JVM when it was launched.  In this diagram, the framework launcher has 
also been migrated to Java 9 in order to have it create a Layer for the 
class loader used to load the framework implementation.  This layer 
configures a single module named system.bundle.  This allows all the 
classes for the Framework implementation to be associated with the 
system.bundle module.  Next is the bundle layer.  This layer is configured 
to map each bundle class loader to a named module representing the bundle. 
 Finally we have a module layer which uses all the built-in module class 
loaders of Java 9 for JPMS.

My Experiment

Over the past few days I have been reworking my github project (OSGi-JPMS 
layer) to investigate if this approach is possible.  Again, I am trying to 
do this without requiring any modifications to the OSGi framework 
implementation itself and I am using only OSGi specified APIs.  This 
approach uses a bottom up strategy for JPMS modules.  With that in mind 
the first thing to do is to modify the OSGi Framework launcher to create 
the system.bundle module.

The system.bundle Module

I did not want to modify the framework itself to make it a real JPMS 
module.  Instead I decided to modify the existing Equinox launcher to 
create a layer itself which maps the class loader it creates to load the 
OSGi Framework with a system.bundle module.  While the Equinox launcher is 
specific to launching the Equinox Framework a similar thing could be done 
to launch any standard OSGi Framework.

The system.bundle acts as the OSGi bundle that exports for all the 
non-java.* packages available in the boot layer.  This allows OSGi bundles 
to use Import-Package to depend on packages from the boot layer.  In order 
to grant the system.bundle class loader access to all packages available 
from the boot layer I have to generate a ModuleDescriptor programatically 
which requires all modules from the boot layer.  The layer must be created 
with the system.bundle module resolved which maps the module to the class 
loader used to load the framework implementation before any classes are 
defined in packages that we want to be exported by system.bundle module. 
The ModuleDescriptor used for the system.bundle must specify that it 
exports the packages from the framework implementation, otherwise JPMS 
will still associate them with the unknown module. With this modified 
launcher, any classes defined in the packages we declared in the 
ModuleDescriptor will be associated with the system.bundle module.  You 
can find the changes I made to the equinox launder on github at 
https://github.com/tjwatson/rt.equinox.framework/tree/tjwatson/jpms.  You 
may notice I hard coded the list of packages to export from the 
system.bundle module.  This was a hack to get going quickly.  Ideally 
these packages would be discovered programmatically.

The Bundle Layer

One important detail to understand about JPMS layers is that the class 
loaders that are mapped to by the modules within a layer MUST NOT have 
defined any classes in packages for which a ModuleDescriptor declares as 
exports or conceals.  This implies that the bundle layer used to represent 
bundle JPMS modules must be created as early as possible and ideally 
before any classes are loaded using the bundle class loaders.  In order to 
achieve this I changed the bundle osgi.jpms.layer to a system.bundle 
fragment still named osgi.jpms.layer.  The OSGi R6 Framework specification 
added a new feature which allows system.bundle fragments to be activated 
when the Framework is initializing before the rest of the bundles get 
activated.  This allows for the code controlling the bundle layer to get 
in place in order to intercept any class defines from bundle class 
loaders.  That way we can map the bundle class loaders for resolved 
bundles to their respective JPMS modules before any classes are defined. I 
used a WovenClassListener and WeavingHook to achieve this.  Here I am not 
interested in actually weaving any class bytes, but these OSGi hooks allow 
for us to hook directly into the bundle class loader just before it is 
about to define a class.

We can now insert the code in the correct place to create the bundle 
layer.  I used a similar approach as before to achieve this, but some more 
information is needed now that the bundle classes will belong to a named 
module.  Here are the steps:

    1. Discover all resolved host bundles and map their symbolic name to 
their wiring.  Note that we could get conflicts if multiple bundles are 
installed with the same symbolic name.  For this experiment I choose only 
one to map into the bundle layer.

    2. Create a module finder that is backed by the bundle wirings.  The 
finder is what creates the ModuleReference and ModuleDescriptor objects to 
represent the bundles.  The following information is used from the wiring:
        - The bundle symbolic name is the module name.
        - The bundle version is the module version.
        - The the package capabilities are the exports for the module.
        - Private packages must be discovered to specify the module's 
concealed packages.  Here the private packages are treated as exported by 
the module instead of concealed.  I will explain why later.
        - Dependencies on other bundles for class loading must become 
module requirements.

    3. Create a configuration using the bundle finder.  Default to using 
the system.bundle layer configuration as the parent configuration.

    4. Create a layer that maps each module name to the bundle wiring 
class loader.

Creating this layer exposes some issues with JPMS that make it difficult 
and sometimes impossible to properly represent OSGi bundles as modules.

    1. JPMS-ISSUE-001 - Reflection is used by almost any framework in Java 
and the OSGi Framework is no exception.  In JPMS the JVM will not allow 
reflection to be used on any class that is not known to JPMS as an 
exported package.  Once I successfully got every class from a bundle 
associated with a JPMS module I found that the framework could no longer 
call Class.newInstance() for bundle activator classes contained in 
concealed packages!!  In order to get that to work I had to treat every 
private package from a bundle as exported by the ModuleDescriptor for the 
bundle.  This will also be necessary for other dependency injection 
containers on OSGi, for example, Declarative Services.  I also imagine 
this has to cause issues for other DI containers such as Spring and CDI.

    2. JPMS-ISSUE-002 - Private packages must be discovered and specified 
to JPMS.  As pointed out already, I had to make the private packages 
exported by JPMS, but first I tried to make them concealed.  Either way, 
all packages that are associated with a module must be known to JPMS as 
either exported or concealed.  If this is not done then the classes from 
unknown packages will be associated with the unnamed module.  This places 
an extra burden on the OSGi module system because in OSGi there was no 
reason for the framework to discover the private packages ahead of time.

    3. JPMS-ISSUE-003 - JPMS must be aware of the OSGi bundle dependencies 
for class loader access.  If the module descriptors representing OSGi 
bundles do not declare any module requires then JPMS will not grant the 
read access required to use a class from another module.  The bundle class 
loaders will continue to be able to load the classes from other bundles 
according to import-package and require-bundle rules, but when the class 
is actually used the JVM will throw access exceptions.  This forces us to 
translate the OSGi dependencies into module requires.  If there are 
multiple bundles with the same symbolic name then there is no way to tell 
JPMS which version of the bundle a module depends on.

    4. JPMS-ISSUE-004 - JPMS layers do not allow cycles between modules. 
OSGi bundles are allowed to have cycles.  Since we must make JPMS aware of 
the OSGi bundle dependencies this restricts us to only bundles that have 
no cycles.

    5. JPMS-ISSUE-005 - JPMS layers provide a static module resolution 
graph.  This will prevent OSGi from successfully resolving dynamic package 
imports if they require read access to a new module.

    6. JPMS-ISSUE-006 - JPMS layers allow for multiple versions of the 
same module but it does not appear that modules within that layer or 
contained child layers can influence which version of the module they get 
resolved to.

    7.JPMS-ISSUE-008 - JPMS layers do not allow for split packages.  If 
the OSGi bundles are resolved with split packages then the bundle layer 
cannot be created.

If you can look past these issues we are left with a layer that can 
represent a static set of resolved OSGi bundles as real JPMS modules and 
we can use that layer to create child JPMS layers for loading other JPMS 
modules.

OSGi Bundle Dynamics

The bundle layer we have now represents a static set of resolved OSGi 
bundles in a Framework.  But the bundles in an OSGi Framework are not 
static.  They can be uninstalled, updated, re-resolved, and new bundles 
can be installed.  How can this dynamic nature be represented in JPMS 
layers?  The approach I took was to create a linear graph of layers where 
the youngest child layer represents the current state of the bundles. This 
would look something like this:

[IMAGE REMOVED]

In this scenario we started out with bundle.a and bundle.b resolved in the 
bundle layer 1.  Then we created a module layer 1 to resolve jpms.a and 
jpms.b modules.  Then bundle.b was updated and bundle.c was installed and 
then bundle.b was refreshed in order to flush out its old content and 
class loader.  This leaves bundle layer 1 with a "dead" bundle.b module 
which also makes module layer 1 stale.  So we decide to discard module 
layer 1 and create module layer 2 for jpms.a and jpms.b modules.  To do 
that we need a new bundle layer that represents the current set of 
resolved bundles.

Here we cannot discard bundle layer 1 because it still has at least one 
valid module bundle.a.  We also cannot represent bundle.a module in a new 
layer because we may have already loaded classes from packages contained 
in bundle.a.  Instead of throwing away bundle layer 1 a new bundle layer 2 
is created that uses bundle layer 1 as its parent.  Bundle layer 2 will 
contain all the new versions of modules that are not already represented 
in the parent layers.  This allows the new bundle.b to shadow the "dead" 
bundle.b module in bundle layer 1.  This appears to work.  The only JPMS 
module that cannot be shadowed by a child layer is the java.base module. 
But we are left with a pretty big issue:

    - JPMS-ISSUE-007 - Discarded modules from a JPMS layer will be pinned 
in memory until the complete layer is discarded.  This ultimately leads to 
a huge class loader leak because we cannot properly free up our stale 
bundle class loaders.  It also causes issues for bundles that are 
uninstalled completely.  The "dead" modules for these bundles will 
continue to be available since nothing is shadowing them from child 
layers.  I suppose we could create a empty module that has the same name 
but exports nothing, but that will still allow modules on top to resolve 
when they shouldn't.

Currently the code for the experiment is located in github at 
https://github.com/tjwatson/osgi-jpms-layer/tree/tjwatson/moduleClassLoader 
I did this in the tjwatson/moduleClassLoader branch.


Conclusion

This approach allows for a pretty accurate representation of a static set 
of resolved OSGi bundles as JPMS modules.  But we are left with several 
issues that need to be addressed before this can be considered a truly 
viable solution.  Some may decide these are permanent restrictions of JPMS 
that we will have to live with going forward.  But I believe there are 
some tweaks to JPMS that could go a long ways to making this approach 
close to a complete solution.  Listed below are some changes that would 
help.  I listed them in the order of importance, but I think 1 and 2 are a 
close tie for most important.

    1. Allow for code that manages a JPMS layer to have more control for 
establishing read access for the modules contained in the managed layer. 
The Module addReads method allows for read access to be added for a module 
dynamically at runtime.  But it has a restriction that it must be called 
by a class defined by the module that wants new read access.  It would be 
a great help if we could call addReads from the management code that 
created the layer.  Perhaps an addReads(Module wantsRead, Module toTarget) 
method on Layer that checks the caller module is the same module get 
created the Layer?  This could be used to solve a large set of issues 
outlined above:

        - JPMS-ISSUE-003 - We could avoid having to make JPMS aware of the 
OSGI dependencies if we would be allowed to establish the read access 
ourselves when the bundle layer is created.

        - JPMS-ISSUE-004 - If we avoid having to make JPMS aware of the 
OSGi dependencies then we no longer have worry about restricting cycles.

        - JPMS-ISSUE-005 - If we can dynamically add reads then we can 
enable dynamic package import to work by dynamically adding read access to 
the provider of the package at runtime.

        - JPMS-ISSUE-008 - If we avoid having to make JPMS aware of the 
OSGi dependencies then we no longer have to worry about restricting split 
packages.

    2. Allow for reflection on classes from concealed packages.  Many 
dependency injection containers depend on being able to act upon concealed 
classes in order to construct objects and inject the objects with 
dependencies.  Forcing implementation details to be exported so that these 
classes can be acted upon by DI containers is wrong.

        - JPMS-ISSUE-001 - We would no longer have to declare the bundle 
private packages as exported by the JPMS module.  Instead they can remain 
concealed as they should be.

    3. Allow for sub-graphs of modules to be discarded within a layer.

        - JPMS-ISSUE-007 - This would allow us to flush out the "dead" 
modules which should never be used anymore.

    4. Allow a layer to map a class loader to a default named module.  Any 
classes from unknown packages to the JPMS would be assigned this named 
module instead of the unnamed module.

        - JPMS-ISSUE-002 - This would allow us to avoid having to scan for 
private packages.  Instead we would map the bundle classloader to a module 
and that module could be used for the private packages.

    5. Allow the JPMS requires statement to specify a module version.

        - JPMS-ISSUE-006 - This would allow us to represent multiple 
versions of a bundle within the bundle layer and give JPMS modules the 
ability to specify which version they want.

My hope is that this experiment is useful in providing constructive 
feedback to the JPMS expert group.  I hope they consider enhancing JPMS to 
make JPMS layers more usable with existing module systems like OSGi.

This was extracted from my article at: 
http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html

Tom