Article on JPMS integration with existing module systems

[mark.reinhold at oracle.com]

2016/9/7 6:57:55 -0700, tim_ellison at uk.ibm.com:
> Obviously Tom has come at this integration challenge from his deep
> knowledge of a particular OSGi implementation.  There are some
> important observations, and minor tweaks to the current model that
> will improve JPMS.
>
> Thomas Watson <tjwatson at us.ibm.com> wrote on 18/08/2016 14:45:06:
>> Below is the plain text format from an article I wrote on using JPMS 
>> layers with an existing module system (OSGi).  ... [1]
> 
> ...
> 
>> Conclusion
> 
> Ok, this is where there are a few issues and recommendations that this
> group can debate...
> 
>> This approach allows for a pretty accurate representation of a static set 
>> of resolved OSGi bundles as JPMS modules.  But we are left with several 
>> issues that need to be addressed before this can be considered a truly 
>> viable solution.  Some may decide these are permanent restrictions of JPMS 
>> that we will have to live with going forward.  But I believe there are 
>> some tweaks to JPMS that could go a long ways to making this approach 
>> close to a complete solution.  Listed below are some changes that would 
>> help.  I listed them in the order of importance, but I think 1 and 2 are a 
>> close tie for most important.
>> 
>>    1. Allow for code that manages a JPMS layer to have more control for 
>> establishing read access for the modules contained in the managed layer. 
>> The Module addReads method allows for read access to be added for a module 
>> dynamically at runtime.  But it has a restriction that it must be called 
>> by a class defined by the module that wants new read access.  It would be 
>> a great help if we could call addReads from the management code that 
>> created the layer.  Perhaps an addReads(Module wantsRead, Module toTarget) 
>> method on Layer that checks the caller module is the same module get 
>> created the Layer?  This could be used to solve a large set of issues 
>> outlined above:
> 
> That sounds like a reasonable request.

This has come up in past discussions.  I don't think an API for this is
strictly necessary.  Code that manages a layer presumably controls the
class loaders in that layer, and hence already has the power to load
arbitrary classes.  It can use that power to load tiny synthetic classes
to add additional read edges as required, as Tom realized in his first
experiment.  That's all a bit clunky, however, and we could make it
easier, so for the record I'll enter a new issue:

  #ReadabilityAddedByLayerCreator -- Provide a means by which the code
  that creates a layer can add readability edges from the modules in that
  layer to other modules, whether those modules are in that layer or in
  other layers.

> ...
> 
>>    2. Allow for reflection on classes from concealed packages.  Many 
>> dependency injection containers depend on being able to act upon concealed 
>> classes in order to construct objects and inject the objects with 
>> dependencies.  Forcing implementation details to be exported so that these 
>> classes can be acted upon by DI containers is wrong.
> 
> I believe this is already covered by #ReflectiveAccessToNonExportedTypes
> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes

Yes, I think that's right.

> ...
>
>>    3. Allow for sub-graphs of modules to be discarded within a layer.
> 
> Again, this sounds like a reasonable request.
> 
>>        - JPMS-ISSUE-007 - This would allow us to flush out the "dead" 
>> modules which should never be used anymore.

This is a narrower form of the existing #MutableConfigurations issue [2].
It could be easier to implement, however, so I'll enter it separately as

  #DiscardableModules -- Provide a means by which a sub-graph of modules
  within a layer can be discarded, so that both those modules and their
  class loaders can be reclaimed.

>>    4. Allow a layer to map a class loader to a default named module.  Any 
>> classes from unknown packages to the JPMS would be assigned this named 
>> module instead of the unnamed module.
>> 
>>        - JPMS-ISSUE-002 - This would allow us to avoid having to scan for 
>> private packages.  Instead we would map the bundle classloader to a module 
>> and that module could be used for the private packages.

Rather than introduce a notion of default named modules, I wonder if this
problem could instead be addressed by tooling.  In JPMS we avoid scanning
a modular JAR file for concealed packages by having the `jar` tool add a
list of all the module's packages to the module descriptor as the JAR
file is created.  This list is optional, so if it's missing we do scan
the JAR file, but we expect that to be a rare occurrence.

If I understand the documentation correctly, bndtools does something
similar by adding a `Private-Package` manifest header to record the list
of non-exported packages in a bundle [3].  Could your OSGi-to-JPMS
adaptation use that manifest header, when available, to avoid scanning
for private packages?

>>    5. Allow the JPMS requires statement to specify a module version.
> 
> Consideration of versioning was explicitly excluded from the original
> requirements, and I don't see it coming in at this stage.  If you are
> controlling the resolution of modules then you would be able to take account
> of version numbers in the integration.

Right.  If you create your own ModuleFinder then you can arrange for the
preferred version of each module to appear ahead of any other versions.

>>        - JPMS-ISSUE-006 - This would allow us to represent multiple 
>> versions of a bundle within the bundle layer and give JPMS modules the 
>> ability to specify which version they want.

I think the first part of this, i.e., representing multiple versions of a
bundle within a single layer, can be avoided entirely, as I'll describe
in a separate message.

>> My hope is that this experiment is useful in providing constructive 
>> feedback to the JPMS expert group.  I hope they consider enhancing JPMS to 
>> make JPMS layers more usable with existing module systems like OSGi.
> 
> Most of the suggestions are modest enhancements to the current model, so it
> is encouraging to see that a good level of interoperability can be obtained
> with a few minor tweaks.
> 
> Thanks for doing this experiment Tom.  I look forward to others' comments
> too.

Tom -- thanks again for this detailed feedback; it's been very helpful.

Thinking about your second experiment led me to a slightly different
approach to supporting bidirectional interoperation between OSGi and
JPMS.  It's a big topic, though, so rather than present it here I'll
describe it in a separate message.

- Mark


[1] http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html
[2] http://openjdk.java.net/projects/jigsaw/spec/issues/#MutableConfigurations
[3] http://bnd.bndtools.org/heads/private_package.html