Concerns mapping existing dynamic module systems to a 1-1 Module->Layer

[Thomas Watson]

> From: mark.reinhold at oracle.com
> To: Thomas Watson/Austin/IBM at IBMUS
> Cc: jpms-spec-experts at openjdk.java.net
> Date: 03/08/2017 02:08 PM
> Subject: Re: Concerns mapping existing dynamic module systems to a 
> 1-1 Module->Layer
> 
> 2017/3/8 6:09:11 -0800, tjwatson at us.ibm.com:
> > There have been several topics recently discussing concerns in one 
context 
> > or another when trying to use JPMS with an existing module system 
(like 
> > OSGi or JBoss Modules) where the existing module system supports 
dynamic 
> > modules.
> > 
> > At a high level, the solution involves a 1-1 mapping for 
Module->Layer. 
> >  This allows such a dyamic module system to make use of the dynamic 
Layer 
> > creation to support the existing module system.
> > 
> > From my experience there are some performance concerns that come in 
three 
> > different categories when using a 1-1 mapping between the Module and 
> > Layer.
> > 
> > ...
> > 
> > Altogether the overhead appears to be 4-5 ms per module (on a pretty 
> > snappy machine).  Caching the discovery of private packages does cut 
that 
> > by at least half.  Most of the remaining overhead can be attributed to 
the 
> > fact that we are using 1-1 Module->Layer mappings in order to support 
> > dynamics as well as hiding our own resolution wires that do not fit 
into 
> > the strict rules of JPMS when dealing with split packages or cycles.
> > 
> > Most of the additional methods David Lloyd is suggesting is to get the 

> > functionality needed to do the complete mapping of existing module 
systems 
> > into JPMS.  Essentially I think we are trying to work with JPMS in 
order 
> > to solve the following requirement [1]:
> > 
> > Interoperation — It must be possible for another module system, such 
as 
> > OSGi, to locate Java modules and resolve them using its own resolver, 
> > except possibly for core system modules.
> 
> To clarify, this requirement does not imply bidirectional interoperation
> with dynamic module systems, as we've been discussing here.  All that
> this requirement means is that another module system must be able to 
read
> JPMS module metadata in order to resolve JPMS modules on its own terms.
> In the case of OSGi, e.g., a suitably-enhanced framework implementation
> could use the `java.lang.module.ModuleFinder` API to locate a module's
> definition, and then use the `ModuleReader` API to read its metadata and
> content to synthesize a corresponding OSGi bundle that's loaded by an
> otherwise-ordinary OSGi class loader.

I can confirm that this is possible, I have implemented something along 
these lines.  But I imagine there are going to be JPMS modules out there 
that will not function correctly when their own classes start returning an 
unnamed module from Class.getModule().  In my opinion, for a JPMS module 
to be loaded as a bundle in OSGi the classes for that bundle must be 
associated with a named module.

> 
> Support for bidirectional interoperation is not mandated by the agreed
> requirements nor by the original JSR submission, as I've noted 
previously
> [2].  I thought we'd found a way to achieve this stretch goal for OSGi,
> but this recent discussion suggests that further research and 
exploration
> is required.  I hope you'll understand that I'm reluctant to delay this
> JSR any further in order to accommodate such work.
> 
> > While the solution we have today achieves this to some extent it is 
not 
> > obvious to me that we have done this in the most useful of ways.  In 
> > particular, all the additions David is suggesting are being resisted 
> > because they potentially are exposing internal implementation details 
of 
> > the JVM which is not desired by the JVM team.  To be a complete 
solution 
> > for the JBoss case all (most?) of the methods David suggests are 
needed. 
> >  As I stated already, OSGi really only needs the addPackage and 
> > addExportsToAll methods to be complete.  But this still requires 
double 
> > resolution work to get us back to the functionality we were at with 
Java 8 
> > and that is only if the suggested methods make it into java 9.
> 
> No changes to OSGi implementations, nor to JPMS, should be required in
> order for OSGi implementations to achieve the level of functionality 
that
> they have on Java 8.  An OSGi implementation should run just fine on top
> of Java 9, without interacting directly with JPMS in any way.  (If it
> doesn't then there's a bug somewhere, in which case please report it and
> we'll fix it.)  What we've been discussing here goes well beyond the
> functionality available on Java 8.

Understood.  Let me clarify.  With an enhanced OSGi framework that loads 
OSGi bundles as JPMS modules/layers we cannot achieve the same level of 
OSGi functionality as would be possible when running in the compatibility 
unnamed module space (or on top of Java 8). 

> 
> > Taking a certain interpretation of the above requirement seems to 
imply 
> > that we should be able to have a solution that does not require double 

> > resolution at the JPMS Layer.  It seems to implies that we should be 
able 
> > to use our own resolver to wire the modules for our own module system 
how 
> > ever we see fit.  But I don't think that is really what we got with 
the 
> > current solution which requires JPMS module resolution and then us 
using 
> > the controller to provide the read edges required by our module system 

> > class loaders.  I was not in on the early discussions of JPMS, but was 

> > there any thread of discussion about an approach that provided a 
> > pre-resolved graph to JPMS?
> 
> No, not that I recall.
> 
> >                              It seems like such a solution would 
naturally 
> > fit into the spirit of the interoperation requirement.  I acknowledge 
that 
> > such an idea is a drastic turn in the current design and I don't see 
how 
> > they could be contained in any reasonable way.
> 
> That would indeed be an enormous change at this point in time, and it
> would be justified only by a stretch goal rather than by an agreed
> requirement.
> 
> > I think we are left with the Layer Controller now for better or worse. 
 I 
> > hope the overhead of creating and resolving 100s of Layers can be 
> > optimized in the future so we can avoid having to do this largely 
> > unnecessary resolution work when dealing with existing module systems. 

> >  But if the Controller is what we are left with I think the solution 
needs 
> > to be as complete as possible for the usecases it is used for.  Namely 
to 
> > do the adhering of JBoss Modules and OSGi into JPMS.
> 
> Sorry, but I remain unwilling to expose the requested low-level 
primitive
> operations, for the reasons previously stated.  If the 
`Layer.Controller`
> API is insufficient as it stands today then we should defer support for
> bidirectional interoperation to a future release.

The Layer.Controller gives us the tools to necessary to achieve a form 
bidirectional support.  The Controller should not be dropped from Java 9.

> 
> - Mark
> 
> 
> [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/
> 2015-04-01#interoperation
> [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-
> October/000410.html
> 

Tom