From peter.kriens at aqute.biz  Thu Oct  6 12:16:09 2016
From: peter.kriens at aqute.biz (Peter Kriens)
Date: Thu, 6 Oct 2016 14:16:09 +0200
Subject: Resign from JPMS Expert Group
Message-ID: <42676DF8-513D-4B86-9978-EA95AD9F8151@aQute.biz>

Mark,

I would like to resign from the JSR 376 Expert Group since I have unfortunately too little time to participate on the level that is required.

Since this leaves the EG without OSGi expertise I would like to recommend Neil Bartlett as the ideal candidate. Neil has confirmed that he will be happy to replace me in the JSR Expert Group. I strongly encourage you to accept him because his OSGi expertise will be an excellent addition to the EG.

Kind regards,

	Peter Kriens	



From mark.reinhold at oracle.com  Thu Oct  6 17:06:59 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Thu,  6 Oct 2016 10:06:59 -0700 (PDT)
Subject: Article on JPMS integration with existing module systems
In-Reply-To: <OFB9D82F52.23C5B2F8-ON80258027.0045F544-80258027.004CB66A@notes.na.collabserv.com>
References: <OF349B4469.ABDAB9AA-ON00258013.004B08BD-86258013.004B8BD5@notes.na.collabserv.com>
	<OFB9D82F52.23C5B2F8-ON80258027.0045F544-80258027.004CB66A@notes.na.collabserv.com>
Message-ID: <20161006170659.B2351D38BA@eggemoggin.niobe.net>

2016/9/7 6:57:55 -0700, tim_ellison at uk.ibm.com:
> Obviously Tom has come at this integration challenge from his deep
> knowledge of a particular OSGi implementation.  There are some
> important observations, and minor tweaks to the current model that
> will improve JPMS.
>
> Thomas Watson <tjwatson at us.ibm.com> wrote on 18/08/2016 14:45:06:
>> Below is the plain text format from an article I wrote on using JPMS 
>> layers with an existing module system (OSGi).  ... [1]
> 
> ...
> 
>> Conclusion
> 
> Ok, this is where there are a few issues and recommendations that this
> group can debate...
> 
>> This approach allows for a pretty accurate representation of a static set 
>> of resolved OSGi bundles as JPMS modules.  But we are left with several 
>> issues that need to be addressed before this can be considered a truly 
>> viable solution.  Some may decide these are permanent restrictions of JPMS 
>> that we will have to live with going forward.  But I believe there are 
>> some tweaks to JPMS that could go a long ways to making this approach 
>> close to a complete solution.  Listed below are some changes that would 
>> help.  I listed them in the order of importance, but I think 1 and 2 are a 
>> close tie for most important.
>> 
>>    1. Allow for code that manages a JPMS layer to have more control for 
>> establishing read access for the modules contained in the managed layer. 
>> The Module addReads method allows for read access to be added for a module 
>> dynamically at runtime.  But it has a restriction that it must be called 
>> by a class defined by the module that wants new read access.  It would be 
>> a great help if we could call addReads from the management code that 
>> created the layer.  Perhaps an addReads(Module wantsRead, Module toTarget) 
>> method on Layer that checks the caller module is the same module get 
>> created the Layer?  This could be used to solve a large set of issues 
>> outlined above:
> 
> That sounds like a reasonable request.

This has come up in past discussions.  I don't think an API for this is
strictly necessary.  Code that manages a layer presumably controls the
class loaders in that layer, and hence already has the power to load
arbitrary classes.  It can use that power to load tiny synthetic classes
to add additional read edges as required, as Tom realized in his first
experiment.  That's all a bit clunky, however, and we could make it
easier, so for the record I'll enter a new issue:

  #ReadabilityAddedByLayerCreator -- Provide a means by which the code
  that creates a layer can add readability edges from the modules in that
  layer to other modules, whether those modules are in that layer or in
  other layers.

> ...
> 
>>    2. Allow for reflection on classes from concealed packages.  Many 
>> dependency injection containers depend on being able to act upon concealed 
>> classes in order to construct objects and inject the objects with 
>> dependencies.  Forcing implementation details to be exported so that these 
>> classes can be acted upon by DI containers is wrong.
> 
> I believe this is already covered by #ReflectiveAccessToNonExportedTypes
> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes

Yes, I think that's right.

> ...
>
>>    3. Allow for sub-graphs of modules to be discarded within a layer.
> 
> Again, this sounds like a reasonable request.
> 
>>        - JPMS-ISSUE-007 - This would allow us to flush out the "dead" 
>> modules which should never be used anymore.

This is a narrower form of the existing #MutableConfigurations issue [2].
It could be easier to implement, however, so I'll enter it separately as

  #DiscardableModules -- Provide a means by which a sub-graph of modules
  within a layer can be discarded, so that both those modules and their
  class loaders can be reclaimed.

>>    4. Allow a layer to map a class loader to a default named module.  Any 
>> classes from unknown packages to the JPMS would be assigned this named 
>> module instead of the unnamed module.
>> 
>>        - JPMS-ISSUE-002 - This would allow us to avoid having to scan for 
>> private packages.  Instead we would map the bundle classloader to a module 
>> and that module could be used for the private packages.

Rather than introduce a notion of default named modules, I wonder if this
problem could instead be addressed by tooling.  In JPMS we avoid scanning
a modular JAR file for concealed packages by having the `jar` tool add a
list of all the module's packages to the module descriptor as the JAR
file is created.  This list is optional, so if it's missing we do scan
the JAR file, but we expect that to be a rare occurrence.

If I understand the documentation correctly, bndtools does something
similar by adding a `Private-Package` manifest header to record the list
of non-exported packages in a bundle [3].  Could your OSGi-to-JPMS
adaptation use that manifest header, when available, to avoid scanning
for private packages?

>>    5. Allow the JPMS requires statement to specify a module version.
> 
> Consideration of versioning was explicitly excluded from the original
> requirements, and I don't see it coming in at this stage.  If you are
> controlling the resolution of modules then you would be able to take account
> of version numbers in the integration.

Right.  If you create your own ModuleFinder then you can arrange for the
preferred version of each module to appear ahead of any other versions.

>>        - JPMS-ISSUE-006 - This would allow us to represent multiple 
>> versions of a bundle within the bundle layer and give JPMS modules the 
>> ability to specify which version they want.

I think the first part of this, i.e., representing multiple versions of a
bundle within a single layer, can be avoided entirely, as I'll describe
in a separate message.

>> My hope is that this experiment is useful in providing constructive 
>> feedback to the JPMS expert group.  I hope they consider enhancing JPMS to 
>> make JPMS layers more usable with existing module systems like OSGi.
> 
> Most of the suggestions are modest enhancements to the current model, so it
> is encouraging to see that a good level of interoperability can be obtained
> with a few minor tweaks.
> 
> Thanks for doing this experiment Tom.  I look forward to others' comments
> too.

Tom -- thanks again for this detailed feedback; it's been very helpful.

Thinking about your second experiment led me to a slightly different
approach to supporting bidirectional interoperation between OSGi and
JPMS.  It's a big topic, though, so rather than present it here I'll
describe it in a separate message.

- Mark


[1] http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html
[2] http://openjdk.java.net/projects/jigsaw/spec/issues/#MutableConfigurations
[3] http://bnd.bndtools.org/heads/private_package.html

From mark.reinhold at oracle.com  Thu Oct  6 17:07:59 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Thu,  6 Oct 2016 10:07:59 -0700 (PDT)
Subject: Dynamic modules and bidirectional interoperation
Message-ID: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>

The agreed requirements for this JSR mandate a modest degree of dynamic
configuration [1].  They do not mandate _dynamic modules_, i.e., the
fine-grained dynamic configuration or "hot deployment" that is possible
with other module systems, such as OSGi and JBoss, in which individual
modules can be installed, updated, and uninstalled at any time.

The requirements also mandate that it "be possible for another module
system, such as OSGi, to locate Java modules and resolve them using its
own resolver, except possibly for core system modules" [2].  They do not
mandate _bidirectional interoperation_, which in the case of OSGi would
be the ability to arrange for bundles to refer to JPMS modules and also
vice versa, i.e., for JPMS modules to refer to bundles.

As things stand, the agreed dynamic-configuration and interoperation
requirements appear to be satisfied by the proposed design.  We have
several open issues, however, among them #NonHierarchicalLayers,
#MutableConfigurations, and #LazyConfigurationAndInstantiation, arising
from the desire for dynamic modules expressed by David Lloyd.  More
recently, Thomas Watson reported on his attempt to achieve bidirectional
interoperation between OSGi and JPMS [3], in which he devised a way to
for a layer of JPMS modules to reference a layer of OSGi bundles.  For
his approach to be practical, however, he needs to be able to discard
sub-graphs of modules within a layer so that they and their class loaders
can be reclaimed earlier than the layer itself (#DiscardableModules [4]).

At this stage we could proceed to adopt dynamic modules and bidirectional
interoperation as two additional requirements, to be supported directly
by the module system.  Doing so would, however, lead to a very different
and more complex design, with correspondingly complex implementations,
since it would need to provide all of the features mentioned above, and
quite likely more.  As I've argued elsewhere this would, in my judgement,
make it very difficult -- if not impossible -- to achieve the stated
goals of this JSR, most importantly the goal that the resulting module
system be approachable by Java developers of all skill levels.

Fortunately, I don't think we need to go that far.

Rather than move toward a design that supports these requirements
directly we can instead support them with the present design indirectly,
by modeling a dynamic module as a sequence of JPMS modules over time.

This key idea is latent in Watson's OSGi adaptation, in which he models
OSGi bundle wirings -- rather than bundles themselves -- as JPMS modules.
When bundles are installed, updated, or uninstalled then he synthesizes
new JPMS modules for the resultant new wirings and configures them into
a single new JPMS layer, using the new class loaders created by the OSGi
framework itself.  That _bundle layer_ refers to the previous bundle
layer, if any, so that over time a stack of bundle layers builds up in
which the topmost layer always represents the current bundle wirings.

JPMS modules can be configured and resolved in additional layers above
the current OSGi bundle layer.  If a bundle upon which a JPMS module
depends is updated then the layer that contains that JPMS module is
discarded and a new layer is instantiated, configured to refer to the
new topmost bundle layer and (I assume) from the same set of root JPMS
modules as the original.

A big problem with this scheme, as Watson points out, is that the JPMS
modules corresponding to discarded OSGi bundle wirings can be retained
in memory for a long time, along with their class loaders, class objects,
and related data structures.  That's because every bundle layer remains
in the stack of bundle layers, and that stack is not discarded until the
OSGi framework itself is stopped and discarded (JPMS-ISSUE-007 in [3]).

Watson suggests solving this problem by allowing sub-graphs of modules
within a layer to be discarded (#DiscardableModules [4]).  That would,
however, require layers to mutable, whereas today they are not.  That
layers are immutable brings a number of advantages, including a simpler
conceptual model, a stronger degree of reliable configuration, and safer
implementations.

Here, then, is an alternative approach:

  - Map each bundle wiring to a unique JPMS module, configured and
    instantiated in its own unique bundle-wiring layer.  An OSGi bundle
    will then be associated not only with a sequence of bundle wirings
    over time, but also with a sequence of JPMS module descriptors,
    configurations, modules, and layers over time.  A layer can be
    reclaimed after no strongly-reachable layers refer to it, so
    discarded bundle-wiring layers -- and their class loaders -- won't
    be retained unnecessarily.  There's no need to represent multiple
    versions of the same bundle within a single layer, since every
    bundle wiring has its own layer (part of JPMS-ISSUE-006 in [3]).

  - Extend JPMS with #NonHierarchicalLayers, so that a set of JPMS
    modules can be resolved against modules in more than one parent
    layer.  An ordinary JPMS layer can then have multiple bundle-wiring
    layers and ordinary JPMS layers as parents.

To illustrate, here's a diagram that elaborates upon the second diagram
in Watson's note:

    http://cr.openjdk.java.net/~mr/jigsaw/notes/osgi-on-jpms.png

The left-hand side shows a layer containing two ordinary JPMS modules,
jpms.a and jpms.b, resolved against two OSGi bundle-wiring layers, for
bundle.a and bundle.b.  (The blue boxes represent JPMS modules; the
orange boxes are the class loaders created by the OSGi framework.)  The
right-hand side depicts some later time after which bundle.b has been
updated, so its original JPMS module and layer were discarded and a new
module and layer were created for it (bundle.b layer 2).  The JPMS module
layer had the original bundle.b layer as one of its parents so it, too,
was discarded and a new JPMS layer was created with the second bundle.b
layer -- along with the original, unchanged bundle.a layer -- as its
parents.

This implementation of OSGi bundles in terms of JPMS modules leverages
the fact that OSGi already associates a bundle with a sequence of bundle
wirings over time, and each new bundle wiring has its own unique class
loader.  All we are doing here is wrapping each of these class loaders in
its own module and layer, so that ordinary JPMS modules can be configured
to refer to them.  There could be many such layers but these layers are
very lightweight, since their sole function is to associate a single
module with an existing class loader for the purpose of access control.
This approach should generalize to other module systems that support
dynamic modules in a similar fashion, i.e., by creating a new class
loader each time a dynamic module is updated.

There are, no doubt, further details to work out, such as exactly how to
synthesize the module descriptors for bundle wirings so that they can
sensibly be referenced by ordinary JPMS modules.  Cyclic relationships
amongst OSGi bundles and split packages across bundles can't be modeled
directly in module descriptors, but I suspect they can be accommodated by
inserting the necessary additional readability edges after the relevant
layers are instantiated.  Overall this does, however, appear to be a
promising approach to supporting dynamic modules and bidirectional
interoperation without adding undue complexity to the platform module
system itself.

To enable this with the present design we do need #NonHierarchicalLayers;
that's a significant change, but I think it's feasible and will explore
it further.  In order to model the cyclic graphs and split packages that
are possible in other module systems it'd be convenient, but not strictly
necessary, to have #ReadabilityAddedByLayerCreator [5].  We have no need
for #MutableConfigurations, #LazyConfigurationAndInstantiation, or
#DiscardableModules, which I think is a big win for simplicity.

Comments?

- Mark


[1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#dynamic-configuration
[2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation
[3] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-August/000062.html,
    or http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html
[4] http://openjdk.java.net/projects/jigsaw/spec/issues/#DiscardableModules
[5] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReadabilityAddedByLayerCreator

From david.lloyd at redhat.com  Thu Oct  6 17:17:46 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Thu, 6 Oct 2016 12:17:46 -0500
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
Message-ID: <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com>

A quick question...

On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote:
> Cyclic relationships
> amongst OSGi bundles and split packages across bundles can't be modeled
> directly in module descriptors, but I suspect they can be accommodated by
> inserting the necessary additional readability edges after the relevant
> layers are instantiated.

Do read edges added at run time affect the subsequent linkage of classes 
in the affected module? I was under the impression that that was not the 
case.

-- 
- DML

From tjwatson at us.ibm.com  Thu Oct  6 18:22:16 2016
From: tjwatson at us.ibm.com (Thomas Watson)
Date: Thu, 6 Oct 2016 13:22:16 -0500
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com>
Message-ID: <OFD1FB5ABA.D340F568-ON00258044.006441C6-86258044.0064EE1C@notes.na.collabserv.com>

> From: "David M. Lloyd" <david.lloyd at redhat.com>
> A quick question...
> 
> On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote:
> > Cyclic relationships
> > amongst OSGi bundles and split packages across bundles can't be 
modeled
> > directly in module descriptors, but I suspect they can be accommodated 
by
> > inserting the necessary additional readability edges after the 
relevant
> > layers are instantiated.
> 
> Do read edges added at run time affect the subsequent linkage of classes 

> in the affected module? I was under the impression that that was not the 

> case.
> 
> -- 
> - DML
> 

My assumption is no, but it also is not relevant to the scenario where the 
read edges are being added to a Module representing an entity (e.g. 
bundle) from another module system.  In such a case I assume the other 
module system (OSGi or Jboss) is in complete control over the class loader 
implementation and delegation of the resolution graph for the respective 
module system.  For example, the OSGi bundle class loaders will not all of 
a sudden start delegating to other module class loaders because of a new 
edge represented in JPMS.

But I would be interested to know if dynamically adding read edges for the 
built-in class loaders of JMPS Layers affect class loading/linkage 
delegation at runtime.

Tom



From mark.reinhold at oracle.com  Thu Oct  6 18:36:04 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Thu, 06 Oct 2016 11:36:04 -0700
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com>
Message-ID: <20161006113604.775141470eggemoggin.niobe.net>

2016/10/6 10:17:46 -0700, david.lloyd at redhat.com:
> A quick question...
> 
> On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote:
>>                                              ...  Cyclic relationships
>> amongst OSGi bundles and split packages across bundles can't be modeled
>> directly in module descriptors, but I suspect they can be accommodated by
>> inserting the necessary additional readability edges after the relevant
>> layers are instantiated.
> 
> Do read edges added at run time affect the subsequent linkage of classes 
> in the affected module?

Read edges added at run time only affect run-time access-control
checks.  They do not change the configuration previously computed
for the relevant layer.  Subsequent configurations resolved against
that configuration will not take the new read edge into account.

- Mark

From david.lloyd at redhat.com  Thu Oct  6 18:56:01 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Thu, 6 Oct 2016 13:56:01 -0500
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <20161006113604.775141470eggemoggin.niobe.net>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com>
	<20161006113604.775141470eggemoggin.niobe.net>
Message-ID: <87aa8663-99f2-d1ff-ec85-1e3739cb8e4d@redhat.com>

On 10/06/2016 01:36 PM, mark.reinhold at oracle.com wrote:
> 2016/10/6 10:17:46 -0700, david.lloyd at redhat.com:
>> A quick question...
>>
>> On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote:
>>>                                              ...  Cyclic relationships
>>> amongst OSGi bundles and split packages across bundles can't be modeled
>>> directly in module descriptors, but I suspect they can be accommodated by
>>> inserting the necessary additional readability edges after the relevant
>>> layers are instantiated.
>>
>> Do read edges added at run time affect the subsequent linkage of classes
>> in the affected module?
>
> Read edges added at run time only affect run-time access-control
> checks.  They do not change the configuration previously computed
> for the relevant layer.  Subsequent configurations resolved against
> that configuration will not take the new read edge into account.

Then I do not believe that adding read edges alone will be sufficient as 
a resolution to #CyclicDependencies for either OSGi or our use cases, 
unfortunately, or at least I don't see how it could be.
-- 
- DML

From tjwatson at us.ibm.com  Thu Oct  6 19:20:17 2016
From: tjwatson at us.ibm.com (Thomas Watson)
Date: Thu, 6 Oct 2016 14:20:17 -0500
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <87aa8663-99f2-d1ff-ec85-1e3739cb8e4d@redhat.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net><75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com><20161006113604.775141470eggemoggin.niobe.net>
	<87aa8663-99f2-d1ff-ec85-1e3739cb8e4d@redhat.com>
Message-ID: <OFA077B247.F46EF4FD-ON00258044.00688577-86258044.006A3DF4@notes.na.collabserv.com>

> From: "David M. Lloyd" <david.lloyd at redhat.com>
> > On 10/06/2016 01:36 PM, mark.reinhold at oracle.com wrote:
> > 2016/10/6 10:17:46 -0700, david.lloyd at redhat.com:
> >> A quick question...
> >>
> >> On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote:
> >>>                                              ...  Cyclic 
relationships
> >>> amongst OSGi bundles and split packages across bundles can't be 
modeled
> >>> directly in module descriptors, but I suspect they can be 
accommodated by
> >>> inserting the necessary additional readability edges after the 
relevant
> >>> layers are instantiated.
> >>
> >> Do read edges added at run time affect the subsequent linkage of 
classes
> >> in the affected module?
> >
> > Read edges added at run time only affect run-time access-control
> > checks.  They do not change the configuration previously computed
> > for the relevant layer.  Subsequent configurations resolved against
> > that configuration will not take the new read edge into account.
> 
> Then I do not believe that adding read edges alone will be sufficient as 

> a resolution to #CyclicDependencies for either OSGi or our use cases, 
> unfortunately, or at least I don't see how it could be.
> -- 
> - DML
> 

My understanding is that read edges are needed to get past the 
accessibility checks that are done by JPMS when a class is loaded by a 
class loader which is associated with a Module known to JPMS itself.  In 
my experiment I attempted to represent the OSGi class loaders (wirings) 
with a generated module which lives inside a bundle JMPS Layer.  Once I 
did this all the accessibility rules (for execution not class loading) are 
enabled at the JPMS layer according to the read edges known by JPMS.  But 
the class loader delegation is still completely under the control of the 
OSGi module system (framework).  The read edges are needed so we can 
represent the possible cycles, and even access to split packages from the 
OSGi resolution graph between the Modules within the bundle layer(s).  In 
my experiment I chose to not express any dependencies in the 
ModuleDesciptors which represent the OSGi bundles.  Instead I create the 
Layer holding the Modules and then after the Layer is "resolved" from JPMS 
point of view I do addReads to wire the modules up according to the 
delegation rules of OSGi resolution.  This way I can get passed the cycle 
and split package restrictions that are in place during JPMS Layer 
resolution.

Tom.



From david.lloyd at redhat.com  Thu Oct  6 19:30:54 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Thu, 6 Oct 2016 14:30:54 -0500
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <OFA077B247.F46EF4FD-ON00258044.00688577-86258044.006A3DF4@notes.na.collabserv.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com>
	<20161006113604.775141470eggemoggin.niobe.net>
	<87aa8663-99f2-d1ff-ec85-1e3739cb8e4d@redhat.com>
	<OFA077B247.F46EF4FD-ON00258044.00688577-86258044.006A3DF4@notes.na.collabserv.com>
Message-ID: <66e629d2-0378-0058-18c9-e9d2704fd603@redhat.com>

On 10/06/2016 02:20 PM, Thomas Watson wrote:
>
>
>> From: "David M. Lloyd" <david.lloyd at redhat.com>
>> > On 10/06/2016 01:36 PM, mark.reinhold at oracle.com wrote:
>> > 2016/10/6 10:17:46 -0700, david.lloyd at redhat.com:
>> >> A quick question...
>> >>
>> >> On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote:
>> >>>                                              ...  Cyclic relationships
>> >>> amongst OSGi bundles and split packages across bundles can't be modeled
>> >>> directly in module descriptors, but I suspect they can be accommodated by
>> >>> inserting the necessary additional readability edges after the relevant
>> >>> layers are instantiated.
>> >>
>> >> Do read edges added at run time affect the subsequent linkage of classes
>> >> in the affected module?
>> >
>> > Read edges added at run time only affect run-time access-control
>> > checks.  They do not change the configuration previously computed
>> > for the relevant layer.  Subsequent configurations resolved against
>> > that configuration will not take the new read edge into account.
>>
>> Then I do not believe that adding read edges alone will be sufficient as
>> a resolution to #CyclicDependencies for either OSGi or our use cases,
>> unfortunately, or at least I don't see how it could be.
>> --
>> - DML
>>
>
> My understanding is that read edges are needed to get past the
> accessibility checks that are done by JPMS when a class is loaded by a
> class loader which is associated with a Module known to JPMS itself.  In
> my experiment I attempted to represent the OSGi class loaders (wirings)
> with a generated module which lives inside a bundle JMPS Layer.  Once I
> did this all the accessibility rules (for execution not class loading)
> are enabled at the JPMS layer according to the read edges known by JPMS.
>  But the class loader delegation is still completely under the control
> of the OSGi module system (framework).  The read edges are needed so we
> can represent the possible cycles, and even access to split packages
> from the OSGi resolution graph between the Modules within the bundle
> layer(s).  In my experiment I chose to not express any dependencies in
> the ModuleDesciptors which represent the OSGi bundles.  Instead I create
> the Layer holding the Modules and then after the Layer is "resolved"
> from JPMS point of view I do addReads to wire the modules up according
> to the delegation rules of OSGi resolution.  This way I can get passed
> the cycle and split package restrictions that are in place during JPMS
> Layer resolution.

I see; but, doesn't that defeat the security benefits of the JPMS?  If 
the classes defined by the bundles themselves are outside of modules 
(i.e. in a class loader's unnamed module) then they are globally 
accessible via reflection/setAccessible.  If the security requirements 
are worthwhile, and we've all been proceeding on the assumption that 
they are, then it seems to me that this will ensure the obsolescence of 
OSGi (not to mention our deployment models).

This is the crux of many of our issues: if you accept that the JPMS 
security measures are worthwhile, then we either have a situation where 
the user has to choose between security or flexibility, or else we have 
to sacrifice flexibility to get the added security across the board, 
losing features in the process.  I continue to maintain that this is a 
false choice.

-- 
- DML

From tjwatson at us.ibm.com  Thu Oct  6 20:17:41 2016
From: tjwatson at us.ibm.com (Thomas Watson)
Date: Thu, 6 Oct 2016 15:17:41 -0500
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <66e629d2-0378-0058-18c9-e9d2704fd603@redhat.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com>
	<20161006113604.775141470eggemoggin.niobe.net>
	<87aa8663-99f2-d1ff-ec85-1e3739cb8e4d@redhat.com>
	<OFA077B247.F46EF4FD-ON00258044.00688577-86258044.006A3DF4@notes.na.collabserv.com>
	<66e629d2-0378-0058-18c9-e9d2704fd603@redhat.com>
Message-ID: <OFCBB0528D.44566CE9-ON00258044.006D7155-86258044.006F7F14@notes.na.collabserv.com>

"David M. Lloyd" <david.lloyd at redhat.com> wrote on 10/06/2016 02:30:54 PM:
> >
> > My understanding is that read edges are needed to get past the
> > accessibility checks that are done by JPMS when a class is loaded by a
> > class loader which is associated with a Module known to JPMS itself. 
In
> > my experiment I attempted to represent the OSGi class loaders 
(wirings)
> > with a generated module which lives inside a bundle JMPS Layer.  Once 
I
> > did this all the accessibility rules (for execution not class loading)
> > are enabled at the JPMS layer according to the read edges known by 
JPMS.
> >  But the class loader delegation is still completely under the control
> > of the OSGi module system (framework).  The read edges are needed so 
we
> > can represent the possible cycles, and even access to split packages
> > from the OSGi resolution graph between the Modules within the bundle
> > layer(s).  In my experiment I chose to not express any dependencies in
> > the ModuleDesciptors which represent the OSGi bundles.  Instead I 
create
> > the Layer holding the Modules and then after the Layer is "resolved"
> > from JPMS point of view I do addReads to wire the modules up according
> > to the delegation rules of OSGi resolution.  This way I can get passed
> > the cycle and split package restrictions that are in place during JPMS
> > Layer resolution.
> 
> I see; but, doesn't that defeat the security benefits of the JPMS?  If 
> the classes defined by the bundles themselves are outside of modules 
> (i.e. in a class loader's unnamed module) then they are globally 
> accessible via reflection/setAccessible.

I am not trying to define classes using the unnamed module.  My prototype 
was about trying to get out of the unnamed module space for the classes 
defined by OSGi class loaders.  Instead I am mapping an OSGi class loader 
to a real module that I generated within a JPMS Layer API, by using the 
custom class loader mapping features of JPMS.  The read edges are needed 
to continue to grant the runtime accessibility needed for the packages 
exported by the Module in the OSGi Layer to properly mimic the delegation 
rules OSGi resolves in the framework.  This does not open up deep 
reflection by or to the classes defined by OSGi class loaders.  These 
classes still follow the rules of JMPS.  And that is where the issues 
outlined in the #ReflectiveAccessToNonExportedTypes thread pop up. Ideally 
the ModuleDescriptors representing OSGi bundles will only export the 
packages which the OSGi bundle manifest exports (using Export-Package). 
Depending on how the #ReflectiveAccessToNonExportedTypes works out we also 
would likely need to 'expose' all the private packages from OSGi bundles 
for deep reflection by default.  My assumption is that the 'exposed' 
packages are not available for class loader delegation, only for 
reflection.  That is if we want to maintain the behavior of OSGi today 
which allows deep reflection on any types private or public, exported or 
not.

> If the security requirements 
> are worthwhile, and we've all been proceeding on the assumption that 
> they are, then it seems to me that this will ensure the obsolescence of 
> OSGi (not to mention our deployment models).

I don't disagree.  I would hope that JPMS Layer creation is flexible 
enough to allow our existing module systems to be represented in a custom 
Layer of sorts and maintain a high level of functionality which we enjoy 
today on Java 8.

> 
> This is the crux of many of our issues: if you accept that the JPMS 
> security measures are worthwhile, then we either have a situation where 
> the user has to choose between security or flexibility, or else we have 
> to sacrifice flexibility to get the added security across the board, 
> losing features in the process.  I continue to maintain that this is a 
> false choice.

I don't disagree, but if we have a good outcome to 
#ReflectiveAccessToNonExportedTypes which allows us to 'expose' packages 
for deep reflection then we can at least maintain functionality of our 
existing module systems in a JPMS Layer world.  Perhaps over time the 
existing module systems will be enhanced such that deep reflection is not 
needed as much or at least not assumed available by default.  But such a 
change to existing module systems to restrict such things is going to be 
just as disruptive as the #ReflectiveAccessToNonExportedTypes issue is for 
JPMS, so that will not come quickly in my opinion.

Tom



From tjwatson at us.ibm.com  Fri Oct  7 15:47:53 2016
From: tjwatson at us.ibm.com (Thomas Watson)
Date: Fri, 7 Oct 2016 10:47:53 -0500
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
Message-ID: <OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>

Overall I think this approach with #NonHierarchicalLayers is worth 
pursuing.  A couple of observations.

1) Any OSGi dependencies that influence class loader delegation will not 
be declared in the ModuleDescriptors which represent the OSGi bundle 
wirings.  Instead we will depend on the ability to 
#ReadabilityAddedByLayerCreator to wire up the modules within the bundle 
Layer according to OSGi resolution rules.  It has a drawback if there are 
cases where some system code is trying to reason about modular 
dependencies by using the JPMS module API.  I think this will have to be 
acceptable because I don't think we will be able to fully converge on all 
the possible dependency types available in the existing module systems. 
For example, OSGi has Require-Bundle which maps closing to JPMS requires, 
but OSGi also has Import-Package dependencies which cannot be mapped 
properly into JMPS module Layers.

2) I assume #NonHierarchicalLayers will be restricted to form a directed 
acyclic graph.  As such I assume each node in the graph of Layers would be 
visited only once when resolving new Layers on top even if there are 
multiple paths to the layer from the new Layer being resolved.

Tom



"jpms-spec-observers" <jpms-spec-observers-bounces at openjdk.java.net> wrote 
on 10/06/2016 12:07:59 PM:

> From: mark.reinhold at oracle.com
> 
> Here, then, is an alternative approach:
> 
>   - Map each bundle wiring to a unique JPMS module, configured and
>     instantiated in its own unique bundle-wiring layer.  An OSGi bundle
>     will then be associated not only with a sequence of bundle wirings
>     over time, but also with a sequence of JPMS module descriptors,
>     configurations, modules, and layers over time.  A layer can be
>     reclaimed after no strongly-reachable layers refer to it, so
>     discarded bundle-wiring layers -- and their class loaders -- won't
>     be retained unnecessarily.  There's no need to represent multiple
>     versions of the same bundle within a single layer, since every
>     bundle wiring has its own layer (part of JPMS-ISSUE-006 in [3]).
> 
>   - Extend JPMS with #NonHierarchicalLayers, so that a set of JPMS
>     modules can be resolved against modules in more than one parent
>     layer.  An ordinary JPMS layer can then have multiple bundle-wiring
>     layers and ordinary JPMS layers as parents.
> 
> To illustrate, here's a diagram that elaborates upon the second diagram
> in Watson's note:
> 
>     http://cr.openjdk.java.net/~mr/jigsaw/notes/osgi-on-jpms.png
> 
> The left-hand side shows a layer containing two ordinary JPMS modules,
> jpms.a and jpms.b, resolved against two OSGi bundle-wiring layers, for
> bundle.a and bundle.b.  (The blue boxes represent JPMS modules; the
> orange boxes are the class loaders created by the OSGi framework.)  The
> right-hand side depicts some later time after which bundle.b has been
> updated, so its original JPMS module and layer were discarded and a new
> module and layer were created for it (bundle.b layer 2).  The JPMS 
module
> layer had the original bundle.b layer as one of its parents so it, too,
> was discarded and a new JPMS layer was created with the second bundle.b
> layer -- along with the original, unchanged bundle.a layer -- as its
> parents.
> 
> This implementation of OSGi bundles in terms of JPMS modules leverages
> the fact that OSGi already associates a bundle with a sequence of bundle
> wirings over time, and each new bundle wiring has its own unique class
> loader.  All we are doing here is wrapping each of these class loaders 
in
> its own module and layer, so that ordinary JPMS modules can be 
configured
> to refer to them.  There could be many such layers but these layers are
> very lightweight, since their sole function is to associate a single
> module with an existing class loader for the purpose of access control.
> This approach should generalize to other module systems that support
> dynamic modules in a similar fashion, i.e., by creating a new class
> loader each time a dynamic module is updated.
> 
> There are, no doubt, further details to work out, such as exactly how to
> synthesize the module descriptors for bundle wirings so that they can
> sensibly be referenced by ordinary JPMS modules.  Cyclic relationships
> amongst OSGi bundles and split packages across bundles can't be modeled
> directly in module descriptors, but I suspect they can be accommodated 
by
> inserting the necessary additional readability edges after the relevant
> layers are instantiated.  Overall this does, however, appear to be a
> promising approach to supporting dynamic modules and bidirectional
> interoperation without adding undue complexity to the platform module
> system itself.
> 
> To enable this with the present design we do need 
#NonHierarchicalLayers;
> that's a significant change, but I think it's feasible and will explore
> it further.  In order to model the cyclic graphs and split packages that
> are possible in other module systems it'd be convenient, but not 
strictly
> necessary, to have #ReadabilityAddedByLayerCreator [5].  We have no need
> for #MutableConfigurations, #LazyConfigurationAndInstantiation, or
> #DiscardableModules, which I think is a big win for simplicity.
> 
> Comments?
> 
> - Mark
> 
> 
> [1] 
http://openjdk.java.net/projects/jigsaw/spec/reqs/#dynamic-configuration
> [2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation
> [3] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-
> August/000062.html,
>     or 
http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html
> [4] 
http://openjdk.java.net/projects/jigsaw/spec/issues/#DiscardableModules
> [5] http://openjdk.java.net/projects/jigsaw/spec/issues/
> #ReadabilityAddedByLayerCreator
> 



From mark.reinhold at oracle.com  Tue Oct 11 15:14:14 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Tue, 11 Oct 2016 08:14:14 -0700 (PDT)
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &
	#AwkwardStrongEncapsulation: Weak modules & private exports
In-Reply-To: <617353007.2424567.1474033457532.JavaMail.zimbra@u-pem.fr>
References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
	<617353007.2424567.1474033457532.JavaMail.zimbra@u-pem.fr>
Message-ID: <20161011151414.9D5E9D50F5@eggemoggin.niobe.net>

2016/9/16 6:44:17 -0700, forax at univ-mlv.fr:
> Hi all,
> the goals of this proposal is
> - to explicitly mark packages that allow 'deep' reflection,
> - disable 'deep' reflection by default even for exported packages
> - ease the transition by adding a new migration state between automatic
>   module and strong module (weak module).
> 
> so 3 goals with one stone :)

Indeed!

> Have i miss something obvious but the last goal, easing migration is not
> listed in the issue summary ?  While adding a new kind of module to ease
> migration is maybe a good idea, it also add a new kind of modules when
> thinking about the modules and their relationship, which seems a terrible
> idea. We are loosing the KISS principle of the current module system.

I didn't list ease of migration in the issue summary but I didn't think
that necessary, since it's an overarching goal of this entire effort.

I understand your reluctance to introduce what appears to be another kind
of module, but semantically one can read

    weak module foo.bar {
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

as a simple shorthand for the more verbose

    module foo.bar {
        exports private com.foo.bar.model;
        exports private ...;                // for other packages as needed
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

or, if we were to add wildcards, then

    module foo.bar {
        exports private *;
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

The advantage of introducing weak modules as an explicit concept is that
they nicely capture the reliable-configuration part of our story and, by
their very name, explicitly do not promise strong encapsulation.

I think weak modules are, moreover, easier to explain to developers just
starting to work with modules.  Telling someone that they have to type

    module foo.bar {
        exports private *;
    }

in order to make a simple module's packages available for deep reflection
raises all kinds of questions.  What does `exports` mean?  What does
`private` mean?  What does `*` range over?  It's all a bit like telling
them that they need to type

    public class X {
        public static void main(String... args) { ... }
    }

in order to print "Hello, world!".  With weak modules you can simply
explain that a weak module has no strong encapsulation and so will work
whether it has a programmatic API or, as is the case here, it just
contains model classes to be manipulated by a reflective framework.
Strong encapsulation can be added later.

> I fully agree that 'deep' reflection should be disallowed by default and can
> be allowed package by package.  But as Stephen Colebourne, i fail to see why
> enabling 'deep' reflection on a package is related to the export
> keyword. While 'deep' reflection is somewhat related to visibility, it's not

(I assume you mean accessibility here, rather than visibility.)

> only about visibility of a class/method/field only, by example you can change
> the value of a public final field of a public class using setAccessible. In
> my opinion, exposing the API to 'deep' reflection is a different concern as
> exporting a package, thus using 'private' as a modifier on export is a bad
> idea (it also doesn't work well with the notion of restricted export (export
> to ...)).  Adding a new keyword 'expose' as Stephen propose to enable 'deep'
> reflection is a better way to answer to the goal 1 and 2.

Reflective access to the elements of packages, deep or otherwise, is
intimately associated with language/bytecode-level access because the
reflection API has always aimed to be an API-level simulation of what's
possible at the language/bytecode level.  This is an extremely valuable
property, since it makes it easy to reason about the correctness of
reflective code.  (There are, obviously, some exceptions to this analogy,
namely the long-standing `setAccessible` method and, more recently, the
changes we made for #ReflectionWithoutReadability.)  Both the language
and the reflection API are subject to the access-control rules of the
module system, so using different keywords to express how they're made
subject to those rules strikes me as confusing.

I've read Colebourne's proposal [1], and I can see its intuitive appeal
to developers (like us!) who are used to thinking about all these things.
To a new developer it might be easier to explain his `exposes *` than the
above `exports private *`, but I think explaining weak modules is even
easier.  A new developer is likely to hear the "reliable configuration +
strong encapsulation" mantra fairly early on, and weak modules are simply
the former, without the latter.

A couple of other problems I see with Colebourne's approach:

  - The only way to allow an exported API package to be used reflectively
    is to expose it for deep reflection.  That requires typing another
    directive and, worse, makes the internals of that package available
    for deep reflection, which is probably not what was intended.

  - The interactions between `exports`, `exposes`, and the qualified
    forms of these directives turns out to be pretty complicated once you
    work out all the cases.  (We've done this internally, for a somewhat
    similar proposal that allows both `private` and `dynamic` modifiers
    on the `exports` directive.)  They wind up being nearly as bad as the
    existing rules for protected members [2].  I'm not convinced that
    this relatively small bit of expressive power is worth the additional
    complexity.

> About the other proposed items:
>   - using transitive instead of public for require, i fully agree.

Good.

>   - about removing the dynamic exports feature, a package declared as non
>     exported but declared as exposed will have the same meaning, no ? if yes,
>     i agree that the notion dynamic export should be removed.

I'm not sure if you're thinking of my proposal here, or Colebourne's.

In my proposal you can't "expose" a package for reflection without also
exporting it.  If you export a package then it's available for shallow
reflection at run time; if you add the `private` modifier then it's also
available for deep reflection via `setAccessible`.

As I mentioned in one of the notes in the proposal, removing `dynamic`
does remove some expressive power, since you can no longer export a
package only at run time.  This means that an IDE or a compiler cannot
(as easily) prevent you from writing code against a module that contains,
e.g., JPA POJOs that are not also APIs (which most aren't).  I understand
the use case, but the cost of supporting it by retaining the `dynamic`
modifier (or its semantic equivalent) is rather high, as I noted above.
If experience shows that it's really necessary, we could add it in a
future release.

- Mark


[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html
[2] http://docs.oracle.com/javase/specs/jls/se8/html/jls-6.html#jls-6.6.2

From mark.reinhold at oracle.com  Tue Oct 11 15:15:14 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Tue, 11 Oct 2016 08:15:14 -0700 (PDT)
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &
	#AwkwardStrongEncapsulation: Weak modules & private exports
In-Reply-To: <5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com>
References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
	<5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com>
Message-ID: <20161011151514.A04ECD50FA@eggemoggin.niobe.net>

2016/9/16 8:30:41 -0700, david.lloyd at redhat.com:
> After a fairly detailed review of this proposal, we have determined that 
> it is not acceptable to Red Hat in its present form.  I will list the 
> primary problems here, and then I'll start up discussion on jigsaw-dev 
> of several possible solutions that could work for us.  I'll number these 
> in case anyone wants to respond piece-wise.
> 
> #1) The "weak" designation appears to be pejorative
> 
> Under this solution, many existing frameworks and/or the modules which 
> consume them must be marked "weak" in order to work correctly, or else a 
> much more complex module descriptor must be used.  While this seems to 
> be a stepping stone for migration, we find that it rather makes 
> middleware appear as a second-class citizen.

No insult is intended; "weak" is used here purely in a technical sense,
as the obvious antonym of "strong".  Suggestions for a better word for
this concept are welcome.

> #2) Weak modules are, in fact, weak
> 
> The idea of retaining the Java 8 rules for transition modules appears 
> good on the surface, however the problem is that users may be forced to 
> use weak modules indefinitely in order to interoperate with certain 
> existing middleware and libraries, or else be faced with a potentially 
> complex migration.  If the security benefits of "strong" modules are 
> real, then users will be rightly inclined to move from weak modules to 
> strong modules.  This can be severely problematic if existing 
> middleware, libraries, and large applications cannot easily operate as 
> or with "strong" modules or cannot do so in an "easy to learn"/"easy to 
> use" fashion.  This also means that "weak" modules exist only for the 
> purposes of maintaining a transition period - at some point they will 
> inevitably transition from "convenience" to "attractive nuisance".

Weak modules do fall between automatic and strong modules on the overall
migration path but they're not intended, necessarily, to be transitory.
There's nothing wrong with evolving an existing JAR file into a weak
module and stopping there.  There's also nothing wrong with creating a
new weak module from scratch, as one might do for a module that contains
nothing but JPA model classes, and never evolving it into a strong
module -- there's no benefit in doing that, so don't do it.

> ...
> 
> #3) Controlling reflection access is always bound up with controlling 
> exports
> 
> The proposed exports system is simplistic but not exactly simple, yet 
> several useful modes remain left unaccounted-for.  Internally we 
> examined the usefulness of controlling access to public and so-called 
> "deep" reflection in combination with whether or not a member was 
> exported, and we came up with this logical table:
> 
>           +------------------+---------------------------+
>           |                  |        Module is...       |
>           |  Reflection is... +------------+--------------+
>           |                  |  Exported  | Not Exported |
>           +----------------------------------------------+
>           |        Forbidden | Not Useful |    Useful    |
>           +----------------------------------------------+
>           |      Public Only |   Useful   |    Useful    |
>           +----------------------------------------------+
>           | Public & Private |   Useful   |    Useful    |
>           +------------------+------------+--------------+

I'm not sure how to read this table.  Should the cell containing the text
"Module is ..." instead contain "Member is ..."?

By "exported" I'll assume that you mean non-reflective access, i.e., from
the language at compile time and bytecodes at run time.

Do you have example use cases for each of the "Useful" cells?  Some are
obvious, but others are not.

> ...
> 
> #4) There isn't always a container
> 
> The justification for not providing an easy mechanism to restrict 
> private reflective access to a trusted framework is that a container can 
> always rewrite a module or inject classes.  However there isn't always a 
> container present - something which the existence of weak modules 
> acknowledges.  Some frameworks and middleware are meant to be usable 
> with or without a container, and it is undesirable to force these use 
> cases into a weak module situation or to require a strong coupling in 
> the module descriptor in this case.  Some other solution should be found 
> that works both for containers and "regular" applications.

If you're running outside of a container then how important is it to
precisely restrict access to packages meant for deep reflection?  If
you're outside of a container then you have to trust the person (you?)
who sets up your module path and/or class path.  To take the canonical
case of a module that just contains JPA model classes, if you make it a
weak module or else `exports private` the relevant packages without
qualification so that the hibernate.core (or whatever) module can reflect
into it, then the scope for trouble is limited.  Sure, some other module
on the module path or even code on the class path could reflect into your
module when it's not supposed to, but is it worth complicating the module
system just to handle this scenario?

Don't get me wrong: Strong encapsulation is generally a very good thing.
It's not always absolutely necessary, however, and we need to balance the
number of ways in which it can be used against the overall complexity of
the design.

(It looks like there was no #5 in your list.)

> #6) Not all reflection is 100% "friendly", and that's OK
> 
> Serialization frameworks generally need private reflective access to the 
> module which contains the classes being serialized.  For some 
> frameworks, this access will already be adequate.  Some existing 
> frameworks utilize Unsafe to create uninitialized instances.  However 
> other frameworks, including frameworks that seek to comply with the 
> Serialization Specification [1], use ReflectionFactory, as that class 
> provides (exclusively, as far as I know) the ability to acquire a 
> constructor that produces a _partially_-initialized class where only the 
> constructors of superclasses to a certain depth are called.  It is our 
> view that it would be a regression to force such frameworks to use 
> --add-exports or JVMTI, and using Unsafe for this purpose, or continuing 
> to use it, (as I understand it) puts at risk the ability to perform 
> optimizations which depend on entry point control, which I am given to 
> understand are a strong motivator for many of the design constraints 
> that exist on the Jigsaw implementation.  It should be possible for 
> these frameworks to continue to use ReflectionFactory (unsupported 
> though it may be) if suitably privileged (in the security manager 
> sense), as long as all of the module(s) upon which the framework 
> reflects have granted private reflective access to it (in the module 
> declaration sense).

So far as I know, this is already the case.  The `ReflectionFactory` API
is specifically retained in JDK 9 for this very purpose, per JEP 260 [1].
Do you have an example for which this does not work?

> #7) More loose coupling seems necessary and useful
> 
> In order for typical applications to function with modern middleware as 
> modules, without compromising security, it may be necessary to enhance 
> the loose coupling mechanism (uses/provides), or to provide an 
> additional, similar mechanism, which allows a symbolic coupling which 
> would allow modules to declare (in an abstract manner) modules which 
> need to have reflective access to it in a "friendly" manner. 
> Implementation ideas are forthcoming on jigsaw-dev.
> 
> Providing a way within the system to grant public and/or private 
> reflection access in a specific manner, and doing so in a manner which 
> is easy to use and not prejudicial against existing or future middleware 
> and small or large applications which consume such middleware in the 
> Java SE or EE space, is our most basic requirement for satisfactory 
> resolution of this issue.

To confirm my understanding, would it be correct to say that your main
concern here is scenarios in which some of the classes in a user module
are intended to be manipulated reflectively by some trusted framework
module, but that framework module is either not known at compile time or
is otherwise inconvenient to name in the declaration of the user module?

>                            I will follow up on jigsaw-dev with some 
> implementation thoughts and more specific requirements.

Discussions of issues in the proposed design, and specific proposals
for changing the design, belong here on the EG list.  The jigsaw-dev
list is meant primarily for the discussion of issues outside the scope
of the JSR such as implementation details, the design and implementation
of JDK-specific APIs and tools, and feedback and questions arising from
practical use of the prototype implementation.

- Mark


[1] http://openjdk.java.net/jeps/260

From Tim_Ellison at uk.ibm.com  Wed Oct 12 08:45:16 2016
From: Tim_Ellison at uk.ibm.com (Tim Ellison)
Date: Wed, 12 Oct 2016 09:45:16 +0100
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised)
	&	#AwkwardStrongEncapsulation: Weak modules & private exports
In-Reply-To: <20161011151414.9D5E9D50F5@eggemoggin.niobe.net>
References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>	<617353007.2424567.1474033457532.JavaMail.zimbra@u-pem.fr>
	<20161011151414.9D5E9D50F5@eggemoggin.niobe.net>
Message-ID: <OF96184CFC.96B101D1-ON8025804A.002D92C9-8025804A.0030169B@notes.na.collabserv.com>

mark.reinhold at oracle.com wrote on 11/10/2016 16:14:14:
> 2016/9/16 6:44:17 -0700, forax at univ-mlv.fr:
> > Hi all,
> > the goals of this proposal is
> > - to explicitly mark packages that allow 'deep' reflection,
> > - disable 'deep' reflection by default even for exported packages
> > - ease the transition by adding a new migration state between 
automatic
> >   module and strong module (weak module).
> > 
> > so 3 goals with one stone :)
> 
> Indeed!
> 
> > Have i miss something obvious but the last goal, easing migration is 
not
> > listed in the issue summary ?  While adding a new kind of module to 
ease
> > migration is maybe a good idea, it also add a new kind of modules when
> > thinking about the modules and their relationship, which seems a 
terrible
> > idea. We are loosing the KISS principle of the current module system.
> 
> I didn't list ease of migration in the issue summary but I didn't think
> that necessary, since it's an overarching goal of this entire effort.

Overall the concept of weak module does not provide an attractive 
migration 
step for existing module systems because it requires each module to 
"export 
private" all of its packages in order to retain existing behavior with 
respect to reflection.

This will effectively expose all of the internals of the existing module 
as 
if it were API at runtime and at compile/tool time.

It can be argued that even in existing modular systems there is a limited 
set of packages that must be allowed for deep reflection and the wholesale 

enablement of all packages concealed and exported for deep reflection is 
not needed.  But this is not the case when mapping an existing modular 
system's custom class loaders into a JPMS Layer (e.g. OSGi).  In that case 

we are motivated to preserve the existing behavior for the modules that 
represent the pre-existing modules/bundles.

> I understand your reluctance to introduce what appears to be another 
kind
> of module, but semantically one can read
> 
>     weak module foo.bar {
>         requires hibernate.core;
>         requires hibernate.entitymanager;
>     }
> 
> as a simple shorthand for the more verbose
> 
>     module foo.bar {
>         exports private com.foo.bar.model;
>         exports private ...;                // for other packages as 
needed
>         requires hibernate.core;
>         requires hibernate.entitymanager;
>     }
> 
> or, if we were to add wildcards, then
> 
>     module foo.bar {
>         exports private *;
>         requires hibernate.core;
>         requires hibernate.entitymanager;
>     }
> 
> The advantage of introducing weak modules as an explicit concept is that
> they nicely capture the reliable-configuration part of our story and, by
> their very name, explicitly do not promise strong encapsulation.
> 
> I think weak modules are, moreover, easier to explain to developers just
> starting to work with modules.  Telling someone that they have to type
> 
>     module foo.bar {
>         exports private *;
>     }
> 
> in order to make a simple module's packages available for deep 
reflection
> raises all kinds of questions.  What does `exports` mean?  What does
> `private` mean?  What does `*` range over?  It's all a bit like telling
> them that they need to type
> 
>     public class X {
>         public static void main(String... args) { ... }
>     }
> 
> in order to print "Hello, world!".  With weak modules you can simply
> explain that a weak module has no strong encapsulation and so will work
> whether it has a programmatic API or, as is the case here, it just
> contains model classes to be manipulated by a reflective framework.
> Strong encapsulation can be added later.

The declaration of a package as exported for tool-time linkage/compilation 

and normal runtime class loader delegation is orthogonal to the 
declaration 
of a package as available for 'deep' and/or 'shallow' reflection. 

The proposal mandates that 'shallow' reflection is only allowed on 
exported
packages, and any concealed packages are not available for any type of
reflection, so the use of the keyword 'private' also is confusing. 

Here the 'private' keyword is used to allow broader access to an exported 
package which is counter intuitive.  One would think a private export 
would 
be less accessible than a 'public' export, but in general the proposed 
semantics are troublesome regardless of the keywords used.

> > I fully agree that 'deep' reflection should be disallowed by default 
and can
> > be allowed package by package.  But as Stephen Colebourne, i fail to 
see why
> > enabling 'deep' reflection on a package is related to the export
> > keyword. While 'deep' reflection is somewhat related to visibility, 
it's not
> 
> (I assume you mean accessibility here, rather than visibility.)
> 
> > only about visibility of a class/method/field only, by example youcan 
change
> > the value of a public final field of a public class using 
setAccessible. In
> > my opinion, exposing the API to 'deep' reflection is a different 
concern as
> > exporting a package, thus using 'private' as a modifier on export is a 
bad
> > idea (it also doesn't work well with the notion of restricted export 
(export
> > to ...)).  Adding a new keyword 'expose' as Stephen propose to enable 
'deep'
> > reflection is a better way to answer to the goal 1 and 2.
> 
> Reflective access to the elements of packages, deep or otherwise, is
> intimately associated with language/bytecode-level access because the
> reflection API has always aimed to be an API-level simulation of what's
> possible at the language/bytecode level.  This is an extremely valuable
> property, since it makes it easy to reason about the correctness of
> reflective code.  (There are, obviously, some exceptions to this 
analogy,
> namely the long-standing `setAccessible` method and, more recently, the
> changes we made for #ReflectionWithoutReadability.)  Both the language
> and the reflection API are subject to the access-control rules of the
> module system, so using different keywords to express how they're made
> subject to those rules strikes me as confusing.
> 
> I've read Colebourne's proposal [1], and I can see its intuitive appeal
> to developers (like us!) who are used to thinking about all these 
things.
> To a new developer it might be easier to explain his `exposes *` than 
the
> above `exports private *`, but I think explaining weak modules is even
> easier.  A new developer is likely to hear the "reliable configuration +
> strong encapsulation" mantra fairly early on, and weak modules are 
simply
> the former, without the latter.

I disagree.  I like the Colebourne's approach because it keeps to two 
declarations ('exports' and 'exposes') as orthogonal concepts.  However,
I don't see the need to open the syntax up for wildcards '*' at this time.

> A couple of other problems I see with Colebourne's approach:
> 
>   - The only way to allow an exported API package to be used 
reflectively
>     is to expose it for deep reflection.  That requires typing another
>     directive and, worse, makes the internals of that package available
>     for deep reflection, which is probably not what was intended.
> 
>   - The interactions between `exports`, `exposes`, and the qualified
>     forms of these directives turns out to be pretty complicated once 
you
>     work out all the cases.  (We've done this internally, for a somewhat
>     similar proposal that allows both `private` and `dynamic` modifiers
>     on the `exports` directive.)  They wind up being nearly as bad as 
the
>     existing rules for protected members [2].  I'm not convinced that
>     this relatively small bit of expressive power is worth the 
additional
>     complexity.

It would be interesting to see that exploration exposed, because as you 
say
the Colebourne expression appears intuitive.

> > About the other proposed items:
> >   - using transitive instead of public for require, i fully agree.
> 
> Good.
> 
> >   - about removing the dynamic exports feature, a package declared as 
non
> >     exported but declared as exposed will have the same meaning, 
> no ? if yes,
> >     i agree that the notion dynamic export should be removed.
> 
> I'm not sure if you're thinking of my proposal here, or Colebourne's.
> 
> In my proposal you can't "expose" a package for reflection without also
> exporting it.  If you export a package then it's available for shallow
> reflection at run time; if you add the `private` modifier then it's also
> available for deep reflection via `setAccessible`.

As discussed, it would be useful to explore the options for changing the
proposal's constraint that you can't "expose" a package for reflection 
without 
also exporting it.

Regards,
Tim

> As I mentioned in one of the notes in the proposal, removing `dynamic`
> does remove some expressive power, since you can no longer export a
> package only at run time.  This means that an IDE or a compiler cannot
> (as easily) prevent you from writing code against a module that 
contains,
> e.g., JPA POJOs that are not also APIs (which most aren't).  I 
understand
> the use case, but the cost of supporting it by retaining the `dynamic`
> modifier (or its semantic equivalent) is rather high, as I noted above.
> If experience shows that it's really necessary, we could add it in a
> future release.
> 
> - Mark
> 
> 
> [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-
> September/009370.html
> [2] 
http://docs.oracle.com/javase/specs/jls/se8/html/jls-6.html#jls-6.6.2
> 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

From Tim_Ellison at uk.ibm.com  Wed Oct 12 08:48:06 2016
From: Tim_Ellison at uk.ibm.com (Tim Ellison)
Date: Wed, 12 Oct 2016 09:48:06 +0100
Subject: Resign from JPMS Expert Group
In-Reply-To: <42676DF8-513D-4B86-9978-EA95AD9F8151@aQute.biz>
References: <42676DF8-513D-4B86-9978-EA95AD9F8151@aQute.biz>
Message-ID: <OF3936A89B.9252D7C6-ON8025804A.002FB730-8025804A.00305913@notes.na.collabserv.com>

Peter Kriens <peter.kriens at aqute.biz> wrote on 06/10/2016 13:16:09:
> Mark,
> 
> I would like to resign from the JSR 376 Expert Group since I have 
> unfortunately too little time to participate on the level that is 
required.
> 
> Since this leaves the EG without OSGi expertise I would like to 
> recommend Neil Bartlett as the ideal candidate. Neil has confirmed 
> that he will be happy to replace me in the JSR Expert Group. I 
> strongly encourage you to accept him because his OSGi expertise will
> be an excellent addition to the EG.

Peter, I'm sorry that you have not been able to find the time to 
contribute
some of your valuable experience to the JSR.  I've worked with Neil in the
past and concur that he would be a fitting replacement on the EG.
Of course, it is Mark's prerogative to permit the substitution.

Regards,
Tim



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

From mark.reinhold at oracle.com  Wed Oct 12 19:30:04 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Wed, 12 Oct 2016 12:30:04 -0700
Subject: Resign from JPMS Expert Group
In-Reply-To: <42676DF8-513D-4B86-9978-EA95AD9F8151@aQute.biz>
References: <42676DF8-513D-4B86-9978-EA95AD9F8151@aQute.biz>
Message-ID: <20161012123004.820359933eggemoggin.niobe.net>

2016/10/6 5:16:09 -0700, peter.kriens at aqute.biz:
> Mark,
> 
> I would like to resign from the JSR 376 Expert Group since I have
> unfortunately too little time to participate on the level that is
> required.
> 
> Since this leaves the EG without OSGi expertise I would like to
> recommend Neil Bartlett as the ideal candidate. Neil has confirmed
> that he will be happy to replace me in the JSR Expert Group. I
> strongly encourage you to accept him because his OSGi expertise will
> be an excellent addition to the EG.

Peter,

I'm sorry to hear that you no longer have time to participate.
Thanks for your past contributions.  I'll certainly give Neal
Bartlett's nomination due consideration.

- Mark

From forax at univ-mlv.fr  Fri Oct 21 13:24:38 2016
From: forax at univ-mlv.fr (Remi Forax)
Date: Fri, 21 Oct 2016 15:24:38 +0200 (CEST)
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
Message-ID: <1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr>

Hi Mark,
I see an other way to solve the wiring issue,
use VM anonymous classes (via unsafe.defineAnonymousClass) to generate the glue code (as lambda meta factory does to implement the functional interface at runtime) and method handles as wires (once created a method handle do not do any security check anymore, so the requirement to be in a module with specific parents can be lifted). With that, you don't need to generate the wiring code inside a new module and when a new bundle need to be re-wire, you can mutate the wire (the method handle) by using a mutable callsite.

Said differently, loading/unloading an OSGI bundle is equivalent to doing monkey patching in a dynamic language, so there is no reason to not use the tools we provide for dynamic languages to wire OSGI bundles.

so i vote strongly against that proposal mostly because it's too late to change requirements and because some obvious design options to implement OSGI on top of JPMS have not been tested.

regards,
R?mi

----- Mail original -----
> De: "mark reinhold" <mark.reinhold at oracle.com>
> ?: jpms-spec-experts at openjdk.java.net
> Envoy?: Jeudi 6 Octobre 2016 19:07:59
> Objet: Dynamic modules and bidirectional interoperation

> The agreed requirements for this JSR mandate a modest degree of dynamic
> configuration [1].  They do not mandate _dynamic modules_, i.e., the
> fine-grained dynamic configuration or "hot deployment" that is possible
> with other module systems, such as OSGi and JBoss, in which individual
> modules can be installed, updated, and uninstalled at any time.
> 
> The requirements also mandate that it "be possible for another module
> system, such as OSGi, to locate Java modules and resolve them using its
> own resolver, except possibly for core system modules" [2].  They do not
> mandate _bidirectional interoperation_, which in the case of OSGi would
> be the ability to arrange for bundles to refer to JPMS modules and also
> vice versa, i.e., for JPMS modules to refer to bundles.
> 
> As things stand, the agreed dynamic-configuration and interoperation
> requirements appear to be satisfied by the proposed design.  We have
> several open issues, however, among them #NonHierarchicalLayers,
> #MutableConfigurations, and #LazyConfigurationAndInstantiation, arising
> from the desire for dynamic modules expressed by David Lloyd.  More
> recently, Thomas Watson reported on his attempt to achieve bidirectional
> interoperation between OSGi and JPMS [3], in which he devised a way to
> for a layer of JPMS modules to reference a layer of OSGi bundles.  For
> his approach to be practical, however, he needs to be able to discard
> sub-graphs of modules within a layer so that they and their class loaders
> can be reclaimed earlier than the layer itself (#DiscardableModules [4]).
> 
> At this stage we could proceed to adopt dynamic modules and bidirectional
> interoperation as two additional requirements, to be supported directly
> by the module system.  Doing so would, however, lead to a very different
> and more complex design, with correspondingly complex implementations,
> since it would need to provide all of the features mentioned above, and
> quite likely more.  As I've argued elsewhere this would, in my judgement,
> make it very difficult -- if not impossible -- to achieve the stated
> goals of this JSR, most importantly the goal that the resulting module
> system be approachable by Java developers of all skill levels.
> 
> Fortunately, I don't think we need to go that far.
> 
> Rather than move toward a design that supports these requirements
> directly we can instead support them with the present design indirectly,
> by modeling a dynamic module as a sequence of JPMS modules over time.
> 
> This key idea is latent in Watson's OSGi adaptation, in which he models
> OSGi bundle wirings -- rather than bundles themselves -- as JPMS modules.
> When bundles are installed, updated, or uninstalled then he synthesizes
> new JPMS modules for the resultant new wirings and configures them into
> a single new JPMS layer, using the new class loaders created by the OSGi
> framework itself.  That _bundle layer_ refers to the previous bundle
> layer, if any, so that over time a stack of bundle layers builds up in
> which the topmost layer always represents the current bundle wirings.
> 
> JPMS modules can be configured and resolved in additional layers above
> the current OSGi bundle layer.  If a bundle upon which a JPMS module
> depends is updated then the layer that contains that JPMS module is
> discarded and a new layer is instantiated, configured to refer to the
> new topmost bundle layer and (I assume) from the same set of root JPMS
> modules as the original.
> 
> A big problem with this scheme, as Watson points out, is that the JPMS
> modules corresponding to discarded OSGi bundle wirings can be retained
> in memory for a long time, along with their class loaders, class objects,
> and related data structures.  That's because every bundle layer remains
> in the stack of bundle layers, and that stack is not discarded until the
> OSGi framework itself is stopped and discarded (JPMS-ISSUE-007 in [3]).
> 
> Watson suggests solving this problem by allowing sub-graphs of modules
> within a layer to be discarded (#DiscardableModules [4]).  That would,
> however, require layers to mutable, whereas today they are not.  That
> layers are immutable brings a number of advantages, including a simpler
> conceptual model, a stronger degree of reliable configuration, and safer
> implementations.
> 
> Here, then, is an alternative approach:
> 
>  - Map each bundle wiring to a unique JPMS module, configured and
>    instantiated in its own unique bundle-wiring layer.  An OSGi bundle
>    will then be associated not only with a sequence of bundle wirings
>    over time, but also with a sequence of JPMS module descriptors,
>    configurations, modules, and layers over time.  A layer can be
>    reclaimed after no strongly-reachable layers refer to it, so
>    discarded bundle-wiring layers -- and their class loaders -- won't
>    be retained unnecessarily.  There's no need to represent multiple
>    versions of the same bundle within a single layer, since every
>    bundle wiring has its own layer (part of JPMS-ISSUE-006 in [3]).
> 
>  - Extend JPMS with #NonHierarchicalLayers, so that a set of JPMS
>    modules can be resolved against modules in more than one parent
>    layer.  An ordinary JPMS layer can then have multiple bundle-wiring
>    layers and ordinary JPMS layers as parents.
> 
> To illustrate, here's a diagram that elaborates upon the second diagram
> in Watson's note:
> 
>    http://cr.openjdk.java.net/~mr/jigsaw/notes/osgi-on-jpms.png
> 
> The left-hand side shows a layer containing two ordinary JPMS modules,
> jpms.a and jpms.b, resolved against two OSGi bundle-wiring layers, for
> bundle.a and bundle.b.  (The blue boxes represent JPMS modules; the
> orange boxes are the class loaders created by the OSGi framework.)  The
> right-hand side depicts some later time after which bundle.b has been
> updated, so its original JPMS module and layer were discarded and a new
> module and layer were created for it (bundle.b layer 2).  The JPMS module
> layer had the original bundle.b layer as one of its parents so it, too,
> was discarded and a new JPMS layer was created with the second bundle.b
> layer -- along with the original, unchanged bundle.a layer -- as its
> parents.
> 
> This implementation of OSGi bundles in terms of JPMS modules leverages
> the fact that OSGi already associates a bundle with a sequence of bundle
> wirings over time, and each new bundle wiring has its own unique class
> loader.  All we are doing here is wrapping each of these class loaders in
> its own module and layer, so that ordinary JPMS modules can be configured
> to refer to them.  There could be many such layers but these layers are
> very lightweight, since their sole function is to associate a single
> module with an existing class loader for the purpose of access control.
> This approach should generalize to other module systems that support
> dynamic modules in a similar fashion, i.e., by creating a new class
> loader each time a dynamic module is updated.
> 
> There are, no doubt, further details to work out, such as exactly how to
> synthesize the module descriptors for bundle wirings so that they can
> sensibly be referenced by ordinary JPMS modules.  Cyclic relationships
> amongst OSGi bundles and split packages across bundles can't be modeled
> directly in module descriptors, but I suspect they can be accommodated by
> inserting the necessary additional readability edges after the relevant
> layers are instantiated.  Overall this does, however, appear to be a
> promising approach to supporting dynamic modules and bidirectional
> interoperation without adding undue complexity to the platform module
> system itself.
> 
> To enable this with the present design we do need #NonHierarchicalLayers;
> that's a significant change, but I think it's feasible and will explore
> it further.  In order to model the cyclic graphs and split packages that
> are possible in other module systems it'd be convenient, but not strictly
> necessary, to have #ReadabilityAddedByLayerCreator [5].  We have no need
> for #MutableConfigurations, #LazyConfigurationAndInstantiation, or
> #DiscardableModules, which I think is a big win for simplicity.
> 
> Comments?
> 
> - Mark
> 
> 
> [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#dynamic-configuration
> [2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation
> [3]
> http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-August/000062.html,
>    or http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html
> [4] http://openjdk.java.net/projects/jigsaw/spec/issues/#DiscardableModules
> [5]
> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReadabilityAddedByLayerCreator

From david.lloyd at redhat.com  Mon Oct 24 13:46:11 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Mon, 24 Oct 2016 08:46:11 -0500
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &
	#AwkwardStrongEncapsulation: Weak modules & private exports
In-Reply-To: <20161011151514.A04ECD50FA@eggemoggin.niobe.net>
References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
	<5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com>
	<20161011151514.A04ECD50FA@eggemoggin.niobe.net>
Message-ID: <279bd57d-515a-a983-ffbb-6f569c29495e@redhat.com>

On 10/11/2016 10:15 AM, mark.reinhold at oracle.com wrote:
> 2016/9/16 8:30:41 -0700, david.lloyd at redhat.com:
>> After a fairly detailed review of this proposal, we have determined that
>> it is not acceptable to Red Hat in its present form.  I will list the
>> primary problems here, and then I'll start up discussion on jigsaw-dev
>> of several possible solutions that could work for us.  I'll number these
>> in case anyone wants to respond piece-wise.
>>
>> #1) The "weak" designation appears to be pejorative
>>
>> Under this solution, many existing frameworks and/or the modules which
>> consume them must be marked "weak" in order to work correctly, or else a
>> much more complex module descriptor must be used.  While this seems to
>> be a stepping stone for migration, we find that it rather makes
>> middleware appear as a second-class citizen.
>
> No insult is intended; "weak" is used here purely in a technical sense,
> as the obvious antonym of "strong".  Suggestions for a better word for
> this concept are welcome.

It's the concept itself that is problematic, I think.  Really, if I 
understand correctly, the "weak" notion exists merely as shorthand to 
say "grant permission on all packages to all modules for private 
reflection", which is the sort of thing that a user ought to be saying 
explicitly.  Many users will use weak modules without understanding what 
they are; some will come to understand it and immediately press for best 
practices to get rid of them.  I think that regardless of what the 
concept is named, it's going to spell trouble in the end.

Our position is that grants of reflective access could (and perhaps 
should) be separated completely from exports, though as stated 
previously, exporting should imply public reflection access merely 
because it doesn't really make any sense not to do so.

>> #2) Weak modules are, in fact, weak
>>
>> The idea of retaining the Java 8 rules for transition modules appears
>> good on the surface, however the problem is that users may be forced to
>> use weak modules indefinitely in order to interoperate with certain
>> existing middleware and libraries, or else be faced with a potentially
>> complex migration.  If the security benefits of "strong" modules are
>> real, then users will be rightly inclined to move from weak modules to
>> strong modules.  This can be severely problematic if existing
>> middleware, libraries, and large applications cannot easily operate as
>> or with "strong" modules or cannot do so in an "easy to learn"/"easy to
>> use" fashion.  This also means that "weak" modules exist only for the
>> purposes of maintaining a transition period - at some point they will
>> inevitably transition from "convenience" to "attractive nuisance".
>
> Weak modules do fall between automatic and strong modules on the overall
> migration path but they're not intended, necessarily, to be transitory.
> There's nothing wrong with evolving an existing JAR file into a weak
> module and stopping there.  There's also nothing wrong with creating a
> new weak module from scratch, as one might do for a module that contains
> nothing but JPA model classes, and never evolving it into a strong
> module -- there's no benefit in doing that, so don't do it.

I definitely understand the intent, but I don't think intent will truly 
matter for this case as I've said above.

>> #3) Controlling reflection access is always bound up with controlling
>> exports
>>
>> The proposed exports system is simplistic but not exactly simple, yet
>> several useful modes remain left unaccounted-for.  Internally we
>> examined the usefulness of controlling access to public and so-called
>> "deep" reflection in combination with whether or not a member was
>> exported, and we came up with this logical table:
>>
>>           +------------------+---------------------------+
>>           |                  |        Module is...       |
>>           |  Reflection is... +------------+--------------+
>>           |                  |  Exported  | Not Exported |
>>           +----------------------------------------------+
>>           |        Forbidden | Not Useful |    Useful    |
>>           +----------------------------------------------+
>>           |      Public Only |   Useful   |    Useful    |
>>           +----------------------------------------------+
>>           | Public & Private |   Useful   |    Useful    |
>>           +------------------+------------+--------------+
>
> I'm not sure how to read this table.  Should the cell containing the text
> "Module is ..." instead contain "Member is ..."?

Right.

> By "exported" I'll assume that you mean non-reflective access, i.e., from
> the language at compile time and bytecodes at run time.

Yes.

> Do you have example use cases for each of the "Useful" cells?  Some are
> obvious, but others are not.

Sure, I'll go over each one by one with some example use cases.

Forbidden/Not Exported: Obviously the foundation case for 
access-oriented encapsulation, this is the default mode.

Exported modes:

Public Only/Exported: It doesn't make sense to disallow reflection on 
public types and members that can already be accessed directly, so this 
would be the default access mode to imported members when one module 
imports another.

Public & Private/Exported: This is the JDK 8 compatibility mode, but a 
module should have to opt into this mode, and it should be restricted to 
specific targets whenever possible, else the mechanism is essentially 
defeated.

Not exported modes:

Public Only/Not Exported: This is the mode that would be used (for 
example) by DI and ORM frameworks and containers which operate on public 
members.  Normally a module which imports such a framework would grant 
the necessary reflective access to it as well (though a full export is 
in this case unnecessary); the framework wouldn't contain requires 
clauses for modules which consume it.  Like I suggested in my first 
email, a way to "handshake" this agreement seems like it would be 
necessary: the framework would not allow itself to be imported without 
the importing module acknowledging the access in its configuration.

Public & Private/Not Exported: This is the mode that would be used by 
serialization and DI/ORM frameworks and containers which operate on 
private members (e.g. field-level injection).  Like the above though, 
this level of access is usually granted by the module importing the 
framework in question and should be explicitly opted into if possible.

The better situation however is to separate the grant from the export.

>> #4) There isn't always a container
>>
>> The justification for not providing an easy mechanism to restrict
>> private reflective access to a trusted framework is that a container can
>> always rewrite a module or inject classes.  However there isn't always a
>> container present - something which the existence of weak modules
>> acknowledges.  Some frameworks and middleware are meant to be usable
>> with or without a container, and it is undesirable to force these use
>> cases into a weak module situation or to require a strong coupling in
>> the module descriptor in this case.  Some other solution should be found
>> that works both for containers and "regular" applications.
>
> If you're running outside of a container then how important is it to
> precisely restrict access to packages meant for deep reflection?  If
> you're outside of a container then you have to trust the person (you?)
> who sets up your module path and/or class path.  To take the canonical
> case of a module that just contains JPA model classes, if you make it a
> weak module or else `exports private` the relevant packages without
> qualification so that the hibernate.core (or whatever) module can reflect
> into it, then the scope for trouble is limited.  Sure, some other module
> on the module path or even code on the class path could reflect into your
> module when it's not supposed to, but is it worth complicating the module
> system just to handle this scenario?

In either situation, the person (deployer) who sets up the class path or 
installation can currently do whatever they want.  Nothing in the new 
(or existing) security framework truly makes it possible to run 
untrusted code safely in this regard, nor to safely run trusted code in 
an untrusted environment, nor is it a sufficient solution for secure 
sandboxing even with a security manager present, with or without a 
container.  At best it's a mitigation strategy.

That said: the rise of microservices and reactive programming suggests 
that such standalone (containerless) deployments are, if anything, more 
popular than ever, but they may suffer from security problems which are 
very similar to, if not identical to, those which applications running 
in containers are subject to.  In fact jlink (and the new AOT work for 
that matter) seem particularly and specifically suited to this type of 
use case.

Finally: a JPA model class module is indeed probably harmless in any 
practical security sense, and locking it down is probably not useful 
from a security perspective, but even in this case it may yet be useful 
to restrict package exports such that only _supported_ classes are 
visible (especially when a module is published within an organization 
and contracts must be maintained); i.e. encapsulation is not solely a 
security concern, it also establishes intent as clearly as any interface 
declaration does.

> Don't get me wrong: Strong encapsulation is generally a very good thing.
> It's not always absolutely necessary, however, and we need to balance the
> number of ways in which it can be used against the overall complexity of
> the design.
>
> (It looks like there was no #5 in your list.)

Another genius stroke of... um, well, yeah.

>> #6) Not all reflection is 100% "friendly", and that's OK
>>
>> Serialization frameworks generally need private reflective access to the
>> module which contains the classes being serialized.  For some
>> frameworks, this access will already be adequate.  Some existing
>> frameworks utilize Unsafe to create uninitialized instances.  However
>> other frameworks, including frameworks that seek to comply with the
>> Serialization Specification [1], use ReflectionFactory, as that class
>> provides (exclusively, as far as I know) the ability to acquire a
>> constructor that produces a _partially_-initialized class where only the
>> constructors of superclasses to a certain depth are called.  It is our
>> view that it would be a regression to force such frameworks to use
>> --add-exports or JVMTI, and using Unsafe for this purpose, or continuing
>> to use it, (as I understand it) puts at risk the ability to perform
>> optimizations which depend on entry point control, which I am given to
>> understand are a strong motivator for many of the design constraints
>> that exist on the Jigsaw implementation.  It should be possible for
>> these frameworks to continue to use ReflectionFactory (unsupported
>> though it may be) if suitably privileged (in the security manager
>> sense), as long as all of the module(s) upon which the framework
>> reflects have granted private reflective access to it (in the module
>> declaration sense).
>
> So far as I know, this is already the case.  The `ReflectionFactory` API
> is specifically retained in JDK 9 for this very purpose, per JEP 260 [1].
> Do you have an example for which this does not work?

No, I had just interpreted previous messages as raising the possibility 
of closing this off so I wanted to capture the requirement.  Since this 
is not the case though, that's one less thing to worry about.  The 
recent enhancements to this class have also gone a long way to ease my 
concerns on this.

>> #7) More loose coupling seems necessary and useful
>>
>> In order for typical applications to function with modern middleware as
>> modules, without compromising security, it may be necessary to enhance
>> the loose coupling mechanism (uses/provides), or to provide an
>> additional, similar mechanism, which allows a symbolic coupling which
>> would allow modules to declare (in an abstract manner) modules which
>> need to have reflective access to it in a "friendly" manner.
>> Implementation ideas are forthcoming on jigsaw-dev.
>>
>> Providing a way within the system to grant public and/or private
>> reflection access in a specific manner, and doing so in a manner which
>> is easy to use and not prejudicial against existing or future middleware
>> and small or large applications which consume such middleware in the
>> Java SE or EE space, is our most basic requirement for satisfactory
>> resolution of this issue.
>
> To confirm my understanding, would it be correct to say that your main
> concern here is scenarios in which some of the classes in a user module
> are intended to be manipulated reflectively by some trusted framework
> module, but that framework module is either not known at compile time or
> is otherwise inconvenient to name in the declaration of the user module?

Correct.

We have observed over the past several years that the arrangement and 
structure of modules at deployment/run time generally diverges 
substantially from the structure of modules at compile time.  In order 
for compilation to have any kind of regularity, decoupling API from 
implementation wherever possible seems like a necessity, especially when 
you consider that JSR and other standards will want to be able to 
publish a simple contract for interfacing which is agnostic to 
implementation.  So really you can extend this notion beyond reflection; 
it's just that reflection is a particularly interesting case of the more 
general requirement.

In a practical sense, the most useful unit of coupling between modules 
is ABI.  When you depend on a module, that is really what you're relying 
on: an ABI which adheres to some contract.  Therefore it is useful to 
have names which correspond to ABIs as well as implementations (whether 
this is the same or a separate namespace doesn't really matter from this 
perspective).

>>                            I will follow up on jigsaw-dev with some
>> implementation thoughts and more specific requirements.
>
> Discussions of issues in the proposed design, and specific proposals
> for changing the design, belong here on the EG list.  The jigsaw-dev
> list is meant primarily for the discussion of issues outside the scope
> of the JSR such as implementation details, the design and implementation
> of JDK-specific APIs and tools, and feedback and questions arising from
> practical use of the prototype implementation.
>
> - Mark
>
>
> [1] http://openjdk.java.net/jeps/260
>

-- 
- DML

From tjwatson at us.ibm.com  Mon Oct 24 19:40:06 2016
From: tjwatson at us.ibm.com (Thomas Watson)
Date: Mon, 24 Oct 2016 14:40:06 -0500
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr>
Message-ID: <OF28202619.DFE819DF-ON00258056.0064110A-86258056.006C0D8F@notes.na.collabserv.com>

Can you explain more about how monkey patching would be done in a JVM 
implementation neutral way?  If I understand correctly your suggestion 
would require monkey patching in order to add functionality to the module 
classes provided by the boot layer (Module, Layer, Configuration etc.). 
This additional functionality would allow an OSGi Framework implementation 
to  represent the dynamic set of bundle wirings in an OSGi Framework as 
JPMS modules.

In order to do that it seems we would need extensive knowledge of the 
internal implementation details of the classes we are monkey patching in 
order to modify their internal structures.  I can see how using Unsafe 
would give us the power to hack into the implementation details of jigsaw, 
but I don't view that as a satisfactory solution to the problem.  It would 
bind us to a particular internal implementation of Java 9 in order to work 
correctly.  It places a new dependency on Unsafe class which is from the 
jdk.unsupported module.  I don't think new dependencies on the 
jdk.unsupported module should be advocated.  Doing so has a high 
probability of breaking on a future version of Java if Unsafe is removed 
by default or the implementation details of the module classes change.

Tom



"jpms-spec-observers" <jpms-spec-observers-bounces at openjdk.java.net> wrote 
on 10/21/2016 08:24:38 AM:

> From: Remi Forax <forax at univ-mlv.fr>
> To: mark reinhold <mark.reinhold at oracle.com>
> Cc: jpms-spec-experts at openjdk.java.net
> Date: 10/21/2016 08:25 AM
> Subject: Re: Dynamic modules and bidirectional interoperation
> Sent by: "jpms-spec-observers" 
<jpms-spec-observers-bounces at openjdk.java.net>
> 
> Hi Mark,
> I see an other way to solve the wiring issue,
> use VM anonymous classes (via unsafe.defineAnonymousClass) to 
> generate the glue code (as lambda meta factory does to implement the
> functional interface at runtime) and method handles as wires (once 
> created a method handle do not do any security check anymore, so the
> requirement to be in a module with specific parents can be lifted). 
> With that, you don't need to generate the wiring code inside a new 
> module and when a new bundle need to be re-wire, you can mutate the 
> wire (the method handle) by using a mutable callsite.
> 
> Said differently, loading/unloading an OSGI bundle is equivalent to 
> doing monkey patching in a dynamic language, so there is no reason 
> to not use the tools we provide for dynamic languages to wire OSGI 
bundles.
> 
> so i vote strongly against that proposal mostly because it's too 
> late to change requirements and because some obvious design options 
> to implement OSGI on top of JPMS have not been tested.
> 
> regards,
> R?mi
> 
> ----- Mail original -----
> > De: "mark reinhold" <mark.reinhold at oracle.com>
> > ?: jpms-spec-experts at openjdk.java.net
> > Envoy?: Jeudi 6 Octobre 2016 19:07:59
> > Objet: Dynamic modules and bidirectional interoperation
> 
> > The agreed requirements for this JSR mandate a modest degree of 
dynamic
> > configuration [1].  They do not mandate _dynamic modules_, i.e., the
> > fine-grained dynamic configuration or "hot deployment" that is 
possible
> > with other module systems, such as OSGi and JBoss, in which individual
> > modules can be installed, updated, and uninstalled at any time.
> > 
> > The requirements also mandate that it "be possible for another module
> > system, such as OSGi, to locate Java modules and resolve them using 
its
> > own resolver, except possibly for core system modules" [2].  They do 
not
> > mandate _bidirectional interoperation_, which in the case of OSGi 
would
> > be the ability to arrange for bundles to refer to JPMS modules and 
also
> > vice versa, i.e., for JPMS modules to refer to bundles.
> > 



From mark.reinhold at oracle.com  Tue Oct 25 21:08:49 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Tue, 25 Oct 2016 14:08:49 -0700
Subject: Please welcome Neil Bartlett to JSR 376
Message-ID: <20161025140849.350853791eggemoggin.niobe.net>

Neil Bartlett, a recognized expert on OSGi, has asked to join this EG.

Ordinarily I'm extremely reluctant to accept new EG members this late in
the life of a JSR, long after we've settled the requirements and after
we've been working on a design for over a year.  Peter Kriens's recent
departure, however, leaves the EG with no resident OSGi expertise, and
that's an important gap to fill.

Neil has agreed to accept the requirements as they stand and, further,
acknowledges that our primary goal at this stage is to finish the design
that we have rather than come up with something significantly different.
I'm confident that he'll bring a valuable perspective to our discussions
and help to ensure the overall success of the JSR.

(Neil -- if there's anything further you'd like to say by way of
 introduction, please do so!)

- Mark

From Tim_Ellison at uk.ibm.com  Tue Oct 25 21:22:58 2016
From: Tim_Ellison at uk.ibm.com (Tim Ellison)
Date: Tue, 25 Oct 2016 22:22:58 +0100
Subject: Please welcome Neil Bartlett to JSR 376
In-Reply-To: <20161025140849.350853791eggemoggin.niobe.net>
References: <20161025140849.350853791eggemoggin.niobe.net>
Message-ID: <OFEBEE34B3.675DED73-ON80258057.0075623E-80258057.007575AD@notes.na.collabserv.com>

mark.reinhold at oracle.com wrote on 25/10/2016 22:08:49:
> Neil Bartlett, a recognized expert on OSGi, has asked to join this EG.
> 
> Ordinarily I'm extremely reluctant to accept new EG members this late in
> the life of a JSR, long after we've settled the requirements and after
> we've been working on a design for over a year.  Peter Kriens's recent
> departure, however, leaves the EG with no resident OSGi expertise, and
> that's an important gap to fill.
> 
> Neil has agreed to accept the requirements as they stand and, further,
> acknowledges that our primary goal at this stage is to finish the design
> that we have rather than come up with something significantly different.
> I'm confident that he'll bring a valuable perspective to our discussions
> and help to ensure the overall success of the JSR.
> 
> (Neil -- if there's anything further you'd like to say by way of
>  introduction, please do so!)

Welcome Neil!

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

From david.lloyd at redhat.com  Tue Oct 25 21:49:30 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Tue, 25 Oct 2016 16:49:30 -0500
Subject: Please welcome Neil Bartlett to JSR 376
In-Reply-To: <20161025140849.350853791eggemoggin.niobe.net>
References: <20161025140849.350853791eggemoggin.niobe.net>
Message-ID: <0d21c865-414d-0593-f8c1-0c51bdd416f6@redhat.com>

On 10/25/2016 04:08 PM, mark.reinhold at oracle.com wrote:
> Neil Bartlett, a recognized expert on OSGi, has asked to join this EG.
>
> Ordinarily I'm extremely reluctant to accept new EG members this late in
> the life of a JSR, long after we've settled the requirements and after
> we've been working on a design for over a year.  Peter Kriens's recent
> departure, however, leaves the EG with no resident OSGi expertise, and
> that's an important gap to fill.
>
> Neil has agreed to accept the requirements as they stand and, further,
> acknowledges that our primary goal at this stage is to finish the design
> that we have rather than come up with something significantly different.
> I'm confident that he'll bring a valuable perspective to our discussions
> and help to ensure the overall success of the JSR.
>
> (Neil -- if there's anything further you'd like to say by way of
>  introduction, please do so!)

Welcome aboard, Neil!
-- 
- DML

From mark.reinhold at oracle.com  Wed Oct 26 16:15:45 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Wed, 26 Oct 2016 09:15:45 -0700
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
Message-ID: <20161026091545.152155830eggemoggin.niobe.net>

2016/10/7 8:47:53 -0700, Thomas Watson <tjwatson at us.ibm.com>, jpms-spec-observers:
> Overall I think this approach with #NonHierarchicalLayers is worth 
> pursuing.

Glad to hear it!

>            A couple of observations.
> 
> 1) Any OSGi dependencies that influence class loader delegation will not 
> be declared in the ModuleDescriptors which represent the OSGi bundle 
> wirings.  Instead we will depend on the ability to 
> #ReadabilityAddedByLayerCreator to wire up the modules within the bundle 
> Layer according to OSGi resolution rules.  It has a drawback if there are 
> cases where some system code is trying to reason about modular 
> dependencies by using the JPMS module API.  I think this will have to be 
> acceptable because I don't think we will be able to fully converge on all 
> the possible dependency types available in the existing module systems. 
> For example, OSGi has Require-Bundle which maps closing to JPMS requires, 
> but OSGi also has Import-Package dependencies which cannot be mapped 
> properly into JMPS module Layers.

If by "system code" you mean code in the JDK itself, I don't think this
will be a big problem at run time.  Except for diagnostic purposes,
module descriptors are really only interpreted during resolution and
configuration.  All other run-time reasoning about module readability
is based upon the reads edges in the module graph as reported by the
`java.lang.reflect.Module::canRead` method, and that will include any
reads edges that you add to support bundle wirings.

> 2) I assume #NonHierarchicalLayers will be restricted to form a directed 
> acyclic graph.

Yes.

>                 As such I assume each node in the graph of Layers would be 
> visited only once when resolving new Layers on top even if there are 
> multiple paths to the layer from the new Layer being resolved.

Yes.

- Mark

From mark.reinhold at oracle.com  Wed Oct 26 17:24:47 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Wed, 26 Oct 2016 10:24:47 -0700
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <OF73A10E29.8203919F-ON00258058.0059BBC5-86258058.005AE60E@notes.na.collabserv.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
	<20161026091545.152155830eggemoggin.niobe.net>
	<OF73A10E29.8203919F-ON00258058.0059BBC5-86258058.005AE60E@notes.na.collabserv.com>
Message-ID: <20161026102447.63677664eggemoggin.niobe.net>

2016/10/26 9:32:42 -0700, Thomas Watson <tjwatson at us.ibm.com>, jpms-spec-comments:
> 2016/10/26 09:15:45 -0700, mark.reinhold at oracle.com, jpms-spec-experts:
>> 2016/10/7 8:47:53 -0700, Thomas Watson <tjwatson at us.ibm.com>, jpms-spec-observers:
>>> ...
>>> 
>>> 1) Any OSGi dependencies that influence class loader delegation will not 
>>> be declared in the ModuleDescriptors which represent the OSGi bundle 
>>> wirings.  Instead we will depend on the ability to 
>>> #ReadabilityAddedByLayerCreator to wire up the modules within the bundle 
>>> Layer according to OSGi resolution rules.  It has a drawback if there are 
>>> cases where some system code is trying to reason about modular 
>>> dependencies by using the JPMS module API.  I think this will have to be 
>>> acceptable because I don't think we will be able to fully converge on all 
>>> the possible dependency types available in the existing module systems. 
>>> For example, OSGi has Require-Bundle which maps closing to JPMS requires, 
>>> but OSGi also has Import-Package dependencies which cannot be mapped 
>>> properly into JMPS module Layers.
>> 
>> If by "system code" you mean code in the JDK itself, I don't think this
>> will be a big problem at run time.  Except for diagnostic purposes,
>> module descriptors are really only interpreted during resolution and
>> configuration.  All other run-time reasoning about module readability
>> is based upon the reads edges in the module graph as reported by the
>> `java.lang.reflect.Module::canRead` method, and that will include any
>> reads edges that you add to support bundle wirings.
> 
> Here I am using "system code" in the general sense, but I think any 
> "system code" that cares about wiring will need to pay attention to 
> Module::canRead like you suggest.  I can think of a few frameworks that 
> care about how modules are wired up at runtime.  For example, in OSGi we 
> have the extender pattern. Without going into much detail here, basically 
> an extender implementation module will only act on extendee modules which 
> are wired to the extender module.  If such a pattern exists using JPMS 
> then the "system code" would be the extender implementation.

Okay, I see.

>>> 2) I assume #NonHierarchicalLayers will be restricted to form a directed 
>>> acyclic graph.
>> 
>> Yes.
>> 
>>>                 As such I assume each node in the graph of Layers would be 
>>> visited only once when resolving new Layers on top even if there are 
>>> multiple paths to the layer from the new Layer being resolved.
>> 
>> Yes.
> 
> The motivation for my question was to ask some follow-ons. 
> 
> Will there will be a specified order in which the nodes are traversed
> when there are multiple paths to a node?  More specifically will the
> parents have a specified order in which they are delegated to?

Yes, the parents of a new configuration will be specified in an ordered
list and during resolution they'll be searched depth-first, in the order
in which they occur in the list.  Depth-first order makes it easy to
reason about resolution, since it works just like a search path.

- Mark

From mark.reinhold at oracle.com  Wed Oct 26 20:15:24 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Wed, 26 Oct 2016 13:15:24 -0700
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
Message-ID: <20161026131524.319958580eggemoggin.niobe.net>

2016/10/7 8:47:53 -0700, tjwatson at us.ibm.com:
> Overall I think this approach with #NonHierarchicalLayers is worth 
> pursuing.  A couple of observations.
> 
> 1) Any OSGi dependencies that influence class loader delegation will not 
> be declared in the ModuleDescriptors which represent the OSGi bundle 
> wirings.  Instead we will depend on the ability to 
> #ReadabilityAddedByLayerCreator to wire up the modules within the bundle 
> Layer according to OSGi resolution rules.

A question for you about #ReadabilityAddedByLayerCreator ...

In this approach, will the code that creates a per-bundle layer be
able to specify all of the required reads edges at the time at which
that layer is created?  Or will it need to be able to add reads edges
at one or more later times?

If the answer is the former then the resulting API is likely to be
much simpler.

- Mark

From tjwatson at us.ibm.com  Wed Oct 26 20:46:07 2016
From: tjwatson at us.ibm.com (Thomas Watson)
Date: Wed, 26 Oct 2016 15:46:07 -0500
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <20161026131524.319958580eggemoggin.niobe.net>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
	<20161026131524.319958580eggemoggin.niobe.net>
Message-ID: <OF02B921C3.CC529AD4-ON00258058.00703E17-86258058.00721956@notes.na.collabserv.com>

> From: mark.reinhold at oracle.com
> To: Thomas Watson/Austin/IBM at IBMUS
> Cc: jpms-spec-experts at openjdk.java.net
> Date: 10/26/2016 03:15 PM
> Subject: Re: Dynamic modules and bidirectional interoperation
> 
> 2016/10/7 8:47:53 -0700, tjwatson at us.ibm.com:
> > Overall I think this approach with #NonHierarchicalLayers is worth 
> > pursuing.  A couple of observations.
> > 
> > 1) Any OSGi dependencies that influence class loader delegation will 
not 
> > be declared in the ModuleDescriptors which represent the OSGi bundle 
> > wirings.  Instead we will depend on the ability to 
> > #ReadabilityAddedByLayerCreator to wire up the modules within the 
bundle 
> > Layer according to OSGi resolution rules.
> 
> A question for you about #ReadabilityAddedByLayerCreator ...
> 
> In this approach, will the code that creates a per-bundle layer be
> able to specify all of the required reads edges at the time at which
> that layer is created?  Or will it need to be able to add reads edges
> at one or more later times?
> 
> If the answer is the former then the resulting API is likely to be
> much simpler.
> 
> - Mark
> 

For static wires we could provide the read edges up front at Layer 
creation.  The vast majority of wires would be static, but in OSGi we also 
have something called dynamic package resolution.  Dynamic package 
resolution would happen after layer creation.  Unfortunately the dynamic 
package resolution could result in the need to add a new read edge to a 
previously unknown module.

I suspect there will be an ordering issue for static wires though.  The 
proposal is to have a triplet for each Layer->Module->BundleWiring.  I'm 
not sure what you have in mind for inputs to layer creation, but if the 
static mapping is needed at layer creation (aka Function<String, 
Collection<Module>>) then each Layer->Module->BundleWiring triplet will 
need to be created in a specific order to ensure all required modules are 
created to satisfy the call to the mapping function.  This will also 
prevent the support for cycles which are allowed in OSGi resolution.  I 
would prefer a dynamic API here.

Tom



From mark.reinhold at oracle.com  Wed Oct 26 21:47:41 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Wed, 26 Oct 2016 14:47:41 -0700
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <OF02B921C3.CC529AD4-ON00258058.00703E17-86258058.00721956@notes.na.collabserv.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
	<20161026131524.319958580eggemoggin.niobe.net>
	<OF02B921C3.CC529AD4-ON00258058.00703E17-86258058.00721956@notes.na.collabserv.com>
Message-ID: <20161026144741.136621558eggemoggin.niobe.net>

2016/10/26 13:46:07 -0700, Thomas Watson <tjwatson at us.ibm.com>:
> 2016/10/26 13:15:24 -0700, mark.reinhold at oracle.com:
>> ...
>> 
>> In this approach, will the code that creates a per-bundle layer be
>> able to specify all of the required reads edges at the time at which
>> that layer is created?  Or will it need to be able to add reads edges
>> at one or more later times?
>> 
>> If the answer is the former then the resulting API is likely to be
>> much simpler.
> 
> For static wires we could provide the read edges up front at Layer 
> creation.  The vast majority of wires would be static, but in OSGi we also 
> have something called dynamic package resolution.  Dynamic package 
> resolution would happen after layer creation.  Unfortunately the dynamic 
> package resolution could result in the need to add a new read edge to a 
> previously unknown module.
> 
> I suspect there will be an ordering issue for static wires though.  The 
> proposal is to have a triplet for each Layer->Module->BundleWiring.  I'm 
> not sure what you have in mind for inputs to layer creation, but if the 
> static mapping is needed at layer creation (aka Function<String, 
> Collection<Module>>) then each Layer->Module->BundleWiring triplet will 
> need to be created in a specific order to ensure all required modules are 
> created to satisfy the call to the mapping function.  This will also 
> prevent the support for cycles which are allowed in OSGi resolution.  I 
> would prefer a dynamic API here.

Okay, between dynamic package resolution and readability cycles it seems
that you do need a dynamic API.  Thanks for the quick answer.

- Mark

From mark.reinhold at oracle.com  Thu Oct 27 15:52:36 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Thu, 27 Oct 2016 08:52:36 -0700 (PDT)
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &
	#AwkwardStrongEncapsulation: Weak modules & private exports
In-Reply-To: <OF96184CFC.96B101D1-ON8025804A.002D92C9-8025804A.0030169B@notes.na.collabserv.com>
References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
	<617353007.2424567.1474033457532.JavaMail.zimbra@u-pem.fr>
	<20161011151414.9D5E9D50F5@eggemoggin.niobe.net>
	<OF96184CFC.96B101D1-ON8025804A.002D92C9-8025804A.0030169B@notes.na.collabserv.com>
Message-ID: <20161027155236.D7069DA324@eggemoggin.niobe.net>

2016/10/12 1:45:16 -0700, Tim Ellison <tim_ellison at uk.ibm.com>:
> mark.reinhold at oracle.com wrote on 11/10/2016 16:14:14:
>> ...
>> 
>> I've read Colebourne's proposal [1], and I can see its intuitive appeal
>> to developers (like us!) who are used to thinking about all these things.
>> To a new developer it might be easier to explain his `exposes *` than the
>> above `exports private *`, but I think explaining weak modules is even
>> easier.  A new developer is likely to hear the "reliable configuration +
>> strong encapsulation" mantra fairly early on, and weak modules are simply
>> the former, without the latter.
> 
> I disagree.  I like the Colebourne's approach because it keeps to two 
> declarations ('exports' and 'exposes') as orthogonal concepts.  However,
> I don't see the need to open the syntax up for wildcards '*' at this time.
> 
>> A couple of other problems I see with Colebourne's approach:
>> 
>>  - The only way to allow an exported API package to be used reflectively
>>    is to expose it for deep reflection.  That requires typing another
>>    directive and, worse, makes the internals of that package available
>>    for deep reflection, which is probably not what was intended.

(For the record, Colebourne later pointed out that this is not what he
 intended [1].)

>>  - The interactions between `exports`, `exposes`, and the qualified
>>    forms of these directives turns out to be pretty complicated once you
>>    work out all the cases.  (We've done this internally, for a somewhat
>>    similar proposal that allows both `private` and `dynamic` modifiers
>>    on the `exports` directive.)  They wind up being nearly as bad as the
>>    existing rules for protected members [2].  I'm not convinced that
>>    this relatively small bit of expressive power is worth the additional
>>    complexity.
> 
> It would be interesting to see that exploration exposed, because as you 
> say the Colebourne expression appears intuitive.

On further examination my second statement above turns out not to be
true, especially if you simplify things by avoiding wildcards.

> As discussed, it would be useful to explore the options for changing the
> proposal's constraint that you can't "expose" a package for reflection 
> without also exporting it.

I've written up a new proposal intended to address this which I'll post
in a moment, along with an analysis of the proposals presented thus far.

- Mark


[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-October/009638.html

From mark.reinhold at oracle.com  Thu Oct 27 15:53:36 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Thu, 27 Oct 2016 08:53:36 -0700 (PDT)
Subject: Proposal (revised^2): #ReflectiveAccessToNonExportedTypes &
	#AwkwardStrongEncapsulation: Open modules & open packages
Message-ID: <20161027155336.DBCE7DA329@eggemoggin.niobe.net>

Issue summary
-------------

  #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
  libraries require reflective access to members of the non-exported
  types of other modules; examples include dependency injection (Guice),
  persistence (JPA), debugging tools, code-automation tools, and
  serialization (XStream).  In some cases the particular library to be
  used is not known until run time (e.g., Hibernate and EclipseLink both
  implement JPA).  This capability is also sometimes used to work around
  bugs in unchangeable code.  Access to non-exported packages can, at
  present, only be done via command-line flags, which is extremely
  awkward.  Provide an easier way for reflective code to access such
  non-exported types. [1]

  #AwkwardStrongEncapsulation --- A non-public element of an exported
  package can still be accessed via the `AccessibleObject::setAccessible`
  method of the core reflection API.  The only way to strongly
  encapsulate such an element is to move it to a non-exported package.
  This makes it awkward, at best, to encapsulate the internals of a
  package that defines a public API. [2]

Proposal
--------

(This is the third proposal for #ReflectiveAccessToNonExportedTypes.
 The note that follows [3] summarizes the background and relates the
 alternatives presented thus far.)

Extend the language of module declarations with the concept of _open_
modules.  Open modules make it easy to define modules whose internals
will be accessed at run time by reflection-based frameworks.  Every
package in an open module has the following properties, by default:

  (1) At compile time it is not exported.

  (2) At run time it is exported without regard to split-package
      conflicts or other inconsistencies, i.e., it does not affect
      the operation of the resolver, nor the construction of layer
      configurations, but is simply exported as if by invoking the
      `Module::addExports` method during layer instantiation.

  (3) At run time it is available for _deep reflection_, i.e., all
      of its elements are accessible via the core reflection API
      including non-public elements, which can be accessed via the
      `AccessibleObject::setAccessible` method.

A package in an open module can be exported explicitly, for use at
compile time, via the familiar `exports` directive, so property (1) will
not hold.  In that case it will also partake in run-time resolution, so
property (2) will not hold either.

Suppose we have a module `foo.bar` that contains an API package
`com.foo.bar`, whose public types are intended for use by other modules,
and a non-API package `com.foo.bar.model`, that contains entity classes
to be manipulated by Hibernate via core reflection.  Then the module
declaration

    open module foo.bar {
        exports com.foo.bar;
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

makes all elements, public and otherwise, of all packages available for
deep reflection at run time, but makes only the public and protected
types in `com.foo.bar` accessible at compile time.  Thus Hibernate can
access non-public elements of the `com.foo.bar.model` package via the
`setAccessible` method, and some other module that `requires foo.bar`
can be compiled against the API in the `com.foo.bar` package but not
against the types in any other package.

Open modules can be considered an intermediate step on the migration
path, between automatic modules and normal modules:

  - An automatic module offers the traditional level of encapsulation:
    All packages are both open for deep reflective access and exported
    for ordinary compile-time and run-time access to their public types.

  - An open module offers a modest degree of stronger encapsulation: All
    packages are open for deep reflective access, but the module's author
    must specify which packages, if any, are exported for ordinary
    compile-time and run-time access.

  - A normal module offers the strongest encapsulation: The module's
    author must specify which packages, if any, are open, or exported,
    or both.

An open module is a good starting point for application code that's being
modularized by its author.  It affords a separation of concerns, so that
the author can focus on defining the module's domain-specific API without
having to worry about how the module will be inspected or manipulated at
run time by reflective frameworks.

                                  * * *

To open specific packages in normal module declarations we introduce a
new per-package directive, `opens`.  A package in a normal module can be
opened, or exported, or both, as follows:

  - If a package is only opened (`opens p;`) then properties (1), (2),
    and (3) hold, i.e., its types are not accessible at compile time,
    it does not affect resolution, and all of its elements are available
    for deep reflection at run time.

  - If a package is only exported (`exports p;`) then those properties
    do not hold, and it is exported exactly as it is today.

  - If a package is both exported and opened (`exports p; opens p;`)
    then it is exported as today and is, additionally, available for
    deep reflection at run time (3).

We can use the `opens` directive to refine the previous example so that
only the `com.foo.bar` package is accessible at compile time, only the
`com.foo.bar.model` package is available for deep reflection at run time,
and the non-public elements of all other packages are strongly
encapsulated:

    module foo.bar {
        exports com.foo.bar;
        opens com.foo.bar.model;
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

It's possible to both open and export a package in a normal module, but
it's often inadvisable.  This is especially so for API packages, since
normally an API's internal implementation details should be strongly
encapsulated.  This combination of directives may, however, be useful for
legacy APIs whose internals are known to be accessed by existing code.

To ensure the integrity of the platform we expect that all the modules of
the JDK itself will be normal modules and that very few, if any, packages
will be opened.

The `opens` directive cannot be used in an `open` module, since all
packages in such modules are implicitly open.

                                  * * *

Both the `exports` and `opens` directives can be qualified, so that a
package is exported or opened only to certain other named modules.  As
before, duplicate directives are not permitted in order to ensure easy
readability.  At most one `exports` directive is relevant to any given
package, and at most one `opens` directive is relevant to any given
package.

There is no syntax for wildcards.  If the defaults are not sufficient for
a package then the package must be named explicitly.

                                  * * *

The existing syntax of `requires public` has long been confusing, so we
here take the opportunity to fix that problem by renaming the `public`
modifier in `requires` directives to `transitive`.  Thus the declaration

    module foo.bar {
        exports com.foo.bar;
        requires public java.sql;
    }

is now written

    module foo.bar {
        exports com.foo.bar;
        requires transitive java.sql;
    }

This is potentially confusing in a different way, since in mathematics
the term "transitive" is usually applied to an entire relation rather
than to three specific elements of a set.  Its use here does not, in
particular, mean that the resolver does not interpret plain `requires`
directives when computing the transitive closure of a set of root
modules.  "Transitive" as used here is in the more abstract sense,
expressing the notion of conveying a property -- in this case, the
readability of the required module -- from one thing to another.  On
balance, however, `transitive` does appear superior to `public`.

Notes
-----

  - This proposal is significantly different from both the first and
    second proposals [4][5].  It recognizes that, in practice, it will
    be common to have open modules, all of whose content is available
    for deep reflection but otherwise inaccessible at compile time unless
    explicitly exported.  If finer-grained control over whether a package
    is opened, exported, or both is required then a normal module
    declaration can be used.

  - This proposal is similar to one made by Stephen Colebourne [6],
    in which he uses the term `exposes`, and to various unpublished
    proposals (of which there have been many!).  We here adopt the term
    `opens` rather than `exposes` in order to avoid confusion between
    the phonetically-similar `exports` and `exposes`, especially for
    non-native speakers of English.  We also use `open` to modify the
    `module` keyword rather than introduce wildcards for the `opens`
    directive, since the former is easier to explain.  The latter would,
    further, lead people to expect wildcards for the `exports` directive,
    but we'd prefer to avoid that since it could encourage the careless
    exportation of all of a module's packages.  Exporting a package for
    use as an API should always be an explicit, affirmative choice.

  - A separate `opens` directive, and the avoidance of wildcards,
    simplifies the interactions amongst qualified and unqualified
    package-level directives.  We no longer need complex rules such as
    those shown for the `private` modifier in the previous proposal [5].

  - This proposal implies corresponding changes to the `ModuleDescriptor`
    and `Module` APIs, but these are reasonably straightforward.

  - This proposal amends the proposal for #ResourceEncapsulation [6]
    to replace the use of private exports with open modules and open
    packages.  Wherever that proposal indicates that a resource can be
    located when its effective package name identifies a package that's
    exported privately, assume instead that the resource can be located
    when that package is either an element of an open module or an open
    package in a normal module.

  - If a container is to ensure that a package in an application module
    is available for deep reflection only by a trusted framework module
    then it can arrange for that by rewriting that module's descriptor,
    as suggested previously [8], to insert the appropriate qualified
    `opens` directive.  This works even in the case, pointed out by Jason
    Greene [9], of two modules that open a package of the same name to
    the same framework module, since opened and unexported packages are
    not subject to the usual package-consistency checks.  There is no
    need any longer for the resolution algorithm to take this type of
    scenario into account [a].  A more flexible way to arrange for such
    precisely-constrained access is a separate issue and may be addressed
    by a future proposal.

  - This proposal primarily addresses "friendly" uses of reflection, such
    as dependency injection and persistence, in which the author of a
    module knows in advance that one or more, or possibly all, packages
    must be opened at run time for deep reflective access by frameworks.
    Intrusive access to arbitrary packages of arbitrary modules by, e.g.,
    serialization frameworks or debugging tools, will still require the
    use of sharp knives such as `--add-exports` or `--add-opens`
    command-line options, the legacy unsupported `sun.misc.Unsafe` API
    and related APIs, or JVM TI.

  - Using `--add-exports` or `--add-opens` command-line options, or their
    equivalents, remains awkward, but sometimes they're the only way out.
    To ease migration it's worth considering some way for an application
    packaged as a JAR file to include such options in its `MANIFEST.MF`
    file, as suggested by Simon Nash [b].  This is addressed in the
    proposal for #AddExportsInManifest [c], which is hereby amended to
    rename the `Add-Exports-Private` attribute to `Add-Opens`.


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
[2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
[3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html
[4] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
[5] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000390.html
[6] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000392.html
[7] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html
[8] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
[9] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
[a] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
[b] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
[c] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000391.html

From mark.reinhold at oracle.com  Thu Oct 27 15:54:36 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Thu, 27 Oct 2016 08:54:36 -0700 (PDT)
Subject: Varieties of reflective access
Message-ID: <20161027155436.E080EDA32D@eggemoggin.niobe.net>

This note categorizes the different kinds of inter-module access
operations relevant to #ReflectiveAccessToNonExportedTypes, analyzes the
possible combinations, and then uses those combinations to relate the
presented proposals to each other.

Categories of inter-module access
---------------------------------

Access operations from code in one module to types in another module can
be divided into three categories (for simplicity we ignore the details of
protected elements):

  - _Static access_ is ordinary access at compile time via language
    constructs, and at run time via constant-pool resolution.  (This may
    include access from bytecodes generated at run time.)  Only public
    elements in exported packages can be accessed.  An API package
    intended to be used by other modules should be exported for static
    access.

  - _Shallow reflective access_ is access at run time via the Core
    Reflection API (`java.lang.reflect`) without using `setAccessible`.
    Only public elements in exported packages can be accessed.

  - _Deep reflective access_ is access at run time via the Core
    Reflection API, using the `setAccessible` method to break into
    non-public elements.  Any element, whether public or not, in any
    package, whether exported or not, can be accessed.  Deep reflective
    access implies shallow reflective access.

A key difference between static access and either kind of reflective
access is that, for static access, the module system ensures reliable
configuration.  If the types in a package are accessible statically then
the post-resolution consistency checks are in full force, so there are no
duplicate or conflicting packages.  If only reflective access is provided
then the relevant readability relationships may have been established
only at run time, so there could be split packages and, also, cycles in
the run-time readability graph.

Access via constant-pool resolution at run time without reliable
configuration is possible, but only in the presence of class loaders that
do not delegate to each other in accordance with the readability graph
constructed by the resolver.  The class loaders provided by the module
system itself always delegate correctly.  If custom loaders provided by,
e.g., a container, do not delegate correctly then chaos is likely to
ensue -- and that's not something that the module system can prevent.

Method handles and method-handle lookup objects provide static access,
checked at the point of creation rather than the point of use.  As such,
the notions of shallow vs. deep reflective access don't appear to be
relevant.  In the long term we want to move toward a world in which
clients of frameworks use such lookup objects to grant the access to
internals that is needed by those frameworks.  Whether that's practical
remains to be seen; if it proves not to be so then the shallow vs. deep
dichotomy could be leveraged to govern the creation of the privileged
method handles needed by frameworks that move away from the more brittle
core reflection API.

Combinations & use cases
------------------------

Here are all the combinations of categories, along with some use cases.
A shorter form of this table will be used below to summarize the various
proposals.

    Static  Shallow    Deep    Use cases
      N        N        N      Fully-encapsulated internal packages
      N        N        Y      (N/A)
      N        Y        N      Partly-encapsulated framework clients
      N        Y        Y      Non-encapsulated framework clients
      Y        N        N      API that disallows all reflective access
      Y        N        Y      (N/A)
      Y        Y        N      API that allows shallow reflection
      Y        Y        Y      API that allows deep reflection (= Java SE 8)

Notes and observations:

  - Fully-encapsulated internal packages are fundamental, and are the
    default for packages that are not listed in a module declaration.
    This row ("N N N") will be omitted henceforth.

  - A partly-encapsulated framework client forbids static access and
    allows only shallow reflective access.  This could be used for a
    non-API package whose public POJOs will be manipulated by a framework
    via public methods and fields, while keeping its non-public elements
    strongly encapsulated.  This distinction seems not worth making in
    practice, since reflective frameworks generally override existing
    language-level access controls as needed.  This row ("N Y N") will
    be omitted henceforth.

  - A non-encapsulated framework client forbids static access and allows
    deep reflective access.  This could be used for a non-API package
    whose internals, including private methods and fields, will be
    manipulated by a framework.  This is another fundamental case.

  - The "N/A" rows are labeled as such because deep reflective access
    implies shallow reflective access.  These rows will be omitted
    henceforth.

  - The ability to define an API that disallows all reflective access
    is theoretically interesting but of questionable utility.  This row
    ("Y N N") will be omitted henceforth.

Omitting the aforementioned rows and compressing horizontally, where "S"
in the "Refl?" column means "shallow" and "D" means "deep":

    Stat  Refl?  Use
      N     D    Non-encapsulated framework clients
      Y     S    API that allows shallow reflection
      Y     D    API that allows deep reflection (= Java SE 8)

We now need to decide which combinations are truly useful, and figure out
how to express those combinations in module declarations.  There are many
tradeoffs to be made, the principal one being between expressive power
and approachability.

(1) SotMS, 2016/3/8 [1]
-----------------------

The SotMS document is out of date with respect to the proposals currently
being discussed, but it reflects what's been in the JDK 9 mainline EA
builds for a long time now.

    Stat  Refl?  Declaration
      N     D    N/A
      Y     S    N/A
      Y     D    `exports p`

(2) `exports dynamic`, 2016/6/28 [2]
------------------------------------

    Stat  Refl?  Declaration
      N     D    N/A
      Y     S    N/A
      Y     D    `exports p` OR `exports dynamic p`

This `exports dynamic` construct provides static access, and thus ensures
reliable configuration, but it is ignored at compile time.

(3) Weak modules, 2016/9/12 [3]
-------------------------------

    Stat  Refl?  Declaration
      N     D    N/A
      Y     S    `exports p`
      Y     D    `exports private p` OR `weak module { ... }`

A weak module cannot contain any `exports` directives, and is thus a
useful waypoint on the migration trail.

This proposal was an attempt to simplify (2), by removing `dynamic` and
making weak modules an explicit mechanism rather than a shorthand for
something more complex.  Several commentators have suggested that this is
not just simple but too simple, since a package can be exported for deep
reflection only if it is also exported for static access at compile time.

(4) Colebourne #1
-----------------

    Stat  Refl?  Declaration
      N     D    `exposes p`
      Y     S    `exports p`
      Y     D    `exports p` AND `exposes p`

Colebourne's first proposal [4], as amended by his later comment [5] so
that exported packages are available for shallow reflection.

This proposal allows both `*` wildcards and qualified forms of both the
`exports` and `exposes` directives.  If you use a wildcard with either
directive then you cannot use that directive in any other way.  Wildcards
cannot be used with qualified directives.

(5) Colebourne #2 [5]
---------------------

    Stat  Refl?  Declaration
      N     D    `exposed module { ... }` ONLY
      Y     S    `exports p` (only in non-weak modules)
      Y     D    `weak module { ... }` ONLY

Here Colebourne simplifies his first proposal by removing the ability to
export a single package for anything other than both static access and
shallow reflection.  This does not allow precise control of reflective
access via qualified exports to trusted framework modules.

(6) Open modules & open packages [6]
------------------------------------

    Stat  Refl?  Declaration
      N     D    `opens p` in a `module { ... }`, OR `open module { ... }`
      Y     S    `exports p` in a `module { ... }`
      Y     D    `exports p` AND `opens p`, in a `module { ... }`,
                  OR `exports p` in an `open module { ... }`

This proposal is similar to Colebourne #1 (5), and to various unpublished
proposals.  It uses `opens` instead of `exposes`, `open module { ... }`
rather than wildcards with `exposes`, and does not allow wildcards with
`exports`.


[1] http://openjdk.java.net/projects/jigsaw/spec/sotms/2016-03-08
[2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
[3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000390.html
[4] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html
[5] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-October/009638.html
[6] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html

From forax at univ-mlv.fr  Thu Oct 27 17:04:56 2016
From: forax at univ-mlv.fr (Remi Forax)
Date: Thu, 27 Oct 2016 17:04:56 +0000
Subject: Proposal (revised^2): #ReflectiveAccessToNonExportedTypes
	&	#AwkwardStrongEncapsulation: Open modules & open packages
In-Reply-To: <20161027155336.DBCE7DA329@eggemoggin.niobe.net>
References: <20161027155336.DBCE7DA329@eggemoggin.niobe.net>
Message-ID: <4E125A2F-4C11-4F56-92E7-E08E4041DD04@univ-mlv.fr>

I really like this proposal.

Remi 

On October 27, 2016 5:53:36 PM GMT+02:00, mark.reinhold at oracle.com wrote:
>Issue summary
>-------------
>
>  #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
>  libraries require reflective access to members of the non-exported
> types of other modules; examples include dependency injection (Guice),
>  persistence (JPA), debugging tools, code-automation tools, and
>  serialization (XStream).  In some cases the particular library to be
> used is not known until run time (e.g., Hibernate and EclipseLink both
> implement JPA).  This capability is also sometimes used to work around
>  bugs in unchangeable code.  Access to non-exported packages can, at
>  present, only be done via command-line flags, which is extremely
>  awkward.  Provide an easier way for reflective code to access such
>  non-exported types. [1]
>
>  #AwkwardStrongEncapsulation --- A non-public element of an exported
>package can still be accessed via the `AccessibleObject::setAccessible`
>  method of the core reflection API.  The only way to strongly
>  encapsulate such an element is to move it to a non-exported package.
>  This makes it awkward, at best, to encapsulate the internals of a
>  package that defines a public API. [2]
>
>Proposal
>--------
>
>(This is the third proposal for #ReflectiveAccessToNonExportedTypes.
> The note that follows [3] summarizes the background and relates the
> alternatives presented thus far.)
>
>Extend the language of module declarations with the concept of _open_
>modules.  Open modules make it easy to define modules whose internals
>will be accessed at run time by reflection-based frameworks.  Every
>package in an open module has the following properties, by default:
>
>  (1) At compile time it is not exported.
>
>  (2) At run time it is exported without regard to split-package
>      conflicts or other inconsistencies, i.e., it does not affect
>      the operation of the resolver, nor the construction of layer
>      configurations, but is simply exported as if by invoking the
>      `Module::addExports` method during layer instantiation.
>
>  (3) At run time it is available for _deep reflection_, i.e., all
>      of its elements are accessible via the core reflection API
>      including non-public elements, which can be accessed via the
>      `AccessibleObject::setAccessible` method.
>
>A package in an open module can be exported explicitly, for use at
>compile time, via the familiar `exports` directive, so property (1)
>will
>not hold.  In that case it will also partake in run-time resolution, so
>property (2) will not hold either.
>
>Suppose we have a module `foo.bar` that contains an API package
>`com.foo.bar`, whose public types are intended for use by other
>modules,
>and a non-API package `com.foo.bar.model`, that contains entity classes
>to be manipulated by Hibernate via core reflection.  Then the module
>declaration
>
>    open module foo.bar {
>        exports com.foo.bar;
>        requires hibernate.core;
>        requires hibernate.entitymanager;
>    }
>
>makes all elements, public and otherwise, of all packages available for
>deep reflection at run time, but makes only the public and protected
>types in `com.foo.bar` accessible at compile time.  Thus Hibernate can
>access non-public elements of the `com.foo.bar.model` package via the
>`setAccessible` method, and some other module that `requires foo.bar`
>can be compiled against the API in the `com.foo.bar` package but not
>against the types in any other package.
>
>Open modules can be considered an intermediate step on the migration
>path, between automatic modules and normal modules:
>
>  - An automatic module offers the traditional level of encapsulation:
>    All packages are both open for deep reflective access and exported
>   for ordinary compile-time and run-time access to their public types.
>
> - An open module offers a modest degree of stronger encapsulation: All
>  packages are open for deep reflective access, but the module's author
>    must specify which packages, if any, are exported for ordinary
>    compile-time and run-time access.
>
>  - A normal module offers the strongest encapsulation: The module's
>    author must specify which packages, if any, are open, or exported,
>    or both.
>
>An open module is a good starting point for application code that's
>being
>modularized by its author.  It affords a separation of concerns, so
>that
>the author can focus on defining the module's domain-specific API
>without
>having to worry about how the module will be inspected or manipulated
>at
>run time by reflective frameworks.
>
>                                  * * *
>
>To open specific packages in normal module declarations we introduce a
>new per-package directive, `opens`.  A package in a normal module can
>be
>opened, or exported, or both, as follows:
>
>  - If a package is only opened (`opens p;`) then properties (1), (2),
>    and (3) hold, i.e., its types are not accessible at compile time,
>   it does not affect resolution, and all of its elements are available
>    for deep reflection at run time.
>
>  - If a package is only exported (`exports p;`) then those properties
>    do not hold, and it is exported exactly as it is today.
>
>  - If a package is both exported and opened (`exports p; opens p;`)
>    then it is exported as today and is, additionally, available for
>    deep reflection at run time (3).
>
>We can use the `opens` directive to refine the previous example so that
>only the `com.foo.bar` package is accessible at compile time, only the
>`com.foo.bar.model` package is available for deep reflection at run
>time,
>and the non-public elements of all other packages are strongly
>encapsulated:
>
>    module foo.bar {
>        exports com.foo.bar;
>        opens com.foo.bar.model;
>        requires hibernate.core;
>        requires hibernate.entitymanager;
>    }
>
>It's possible to both open and export a package in a normal module, but
>it's often inadvisable.  This is especially so for API packages, since
>normally an API's internal implementation details should be strongly
>encapsulated.  This combination of directives may, however, be useful
>for
>legacy APIs whose internals are known to be accessed by existing code.
>
>To ensure the integrity of the platform we expect that all the modules
>of
>the JDK itself will be normal modules and that very few, if any,
>packages
>will be opened.
>
>The `opens` directive cannot be used in an `open` module, since all
>packages in such modules are implicitly open.
>
>                                  * * *
>
>Both the `exports` and `opens` directives can be qualified, so that a
>package is exported or opened only to certain other named modules.  As
>before, duplicate directives are not permitted in order to ensure easy
>readability.  At most one `exports` directive is relevant to any given
>package, and at most one `opens` directive is relevant to any given
>package.
>
>There is no syntax for wildcards.  If the defaults are not sufficient
>for
>a package then the package must be named explicitly.
>
>                                  * * *
>
>The existing syntax of `requires public` has long been confusing, so we
>here take the opportunity to fix that problem by renaming the `public`
>modifier in `requires` directives to `transitive`.  Thus the
>declaration
>
>    module foo.bar {
>        exports com.foo.bar;
>        requires public java.sql;
>    }
>
>is now written
>
>    module foo.bar {
>        exports com.foo.bar;
>        requires transitive java.sql;
>    }
>
>This is potentially confusing in a different way, since in mathematics
>the term "transitive" is usually applied to an entire relation rather
>than to three specific elements of a set.  Its use here does not, in
>particular, mean that the resolver does not interpret plain `requires`
>directives when computing the transitive closure of a set of root
>modules.  "Transitive" as used here is in the more abstract sense,
>expressing the notion of conveying a property -- in this case, the
>readability of the required module -- from one thing to another.  On
>balance, however, `transitive` does appear superior to `public`.
>
>Notes
>-----
>
>  - This proposal is significantly different from both the first and
>    second proposals [4][5].  It recognizes that, in practice, it will
>    be common to have open modules, all of whose content is available
>  for deep reflection but otherwise inaccessible at compile time unless
>  explicitly exported.  If finer-grained control over whether a package
>    is opened, exported, or both is required then a normal module
>    declaration can be used.
>
>  - This proposal is similar to one made by Stephen Colebourne [6],
>    in which he uses the term `exposes`, and to various unpublished
>    proposals (of which there have been many!).  We here adopt the term
>    `opens` rather than `exposes` in order to avoid confusion between
>    the phonetically-similar `exports` and `exposes`, especially for
>    non-native speakers of English.  We also use `open` to modify the
>    `module` keyword rather than introduce wildcards for the `opens`
>   directive, since the former is easier to explain.  The latter would,
>  further, lead people to expect wildcards for the `exports` directive,
>    but we'd prefer to avoid that since it could encourage the careless
>    exportation of all of a module's packages.  Exporting a package for
>    use as an API should always be an explicit, affirmative choice.
>
>  - A separate `opens` directive, and the avoidance of wildcards,
>    simplifies the interactions amongst qualified and unqualified
>    package-level directives.  We no longer need complex rules such as
>   those shown for the `private` modifier in the previous proposal [5].
>
>- This proposal implies corresponding changes to the `ModuleDescriptor`
>    and `Module` APIs, but these are reasonably straightforward.
>
>  - This proposal amends the proposal for #ResourceEncapsulation [6]
>    to replace the use of private exports with open modules and open
>    packages.  Wherever that proposal indicates that a resource can be
>    located when its effective package name identifies a package that's
>    exported privately, assume instead that the resource can be located
>    when that package is either an element of an open module or an open
>    package in a normal module.
>
>  - If a container is to ensure that a package in an application module
>    is available for deep reflection only by a trusted framework module
>    then it can arrange for that by rewriting that module's descriptor,
>    as suggested previously [8], to insert the appropriate qualified
>  `opens` directive.  This works even in the case, pointed out by Jason
>    Greene [9], of two modules that open a package of the same name to
>    the same framework module, since opened and unexported packages are
>    not subject to the usual package-consistency checks.  There is no
>    need any longer for the resolution algorithm to take this type of
>    scenario into account [a].  A more flexible way to arrange for such
>  precisely-constrained access is a separate issue and may be addressed
>    by a future proposal.
>
>- This proposal primarily addresses "friendly" uses of reflection, such
>    as dependency injection and persistence, in which the author of a
>    module knows in advance that one or more, or possibly all, packages
>   must be opened at run time for deep reflective access by frameworks.
>  Intrusive access to arbitrary packages of arbitrary modules by, e.g.,
>    serialization frameworks or debugging tools, will still require the
>    use of sharp knives such as `--add-exports` or `--add-opens`
>    command-line options, the legacy unsupported `sun.misc.Unsafe` API
>    and related APIs, or JVM TI.
>
>- Using `--add-exports` or `--add-opens` command-line options, or their
>  equivalents, remains awkward, but sometimes they're the only way out.
>   To ease migration it's worth considering some way for an application
>    packaged as a JAR file to include such options in its `MANIFEST.MF`
>    file, as suggested by Simon Nash [b].  This is addressed in the
>    proposal for #AddExportsInManifest [c], which is hereby amended to
>    rename the `Add-Exports-Private` attribute to `Add-Opens`.
>
>
>[1]
>http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
>[2]
>http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
>[3]
>http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html
>[4]
>http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
>[5]
>http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000390.html
>[6]
>http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000392.html
>[7]
>http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html
>[8]
>http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
>[9]
>http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
>[a]
>http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
>[b]
>http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
>[c]
>http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000391.html

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

From tjwatson at us.ibm.com  Thu Oct 27 17:49:41 2016
From: tjwatson at us.ibm.com (Thomas Watson)
Date: Thu, 27 Oct 2016 12:49:41 -0500
Subject: Proposal (revised^2):
	#ReflectiveAccessToNonExportedTypes&#AwkwardStrongEncapsulation: Open
	modules & open packages
In-Reply-To: <4E125A2F-4C11-4F56-92E7-E08E4041DD04@univ-mlv.fr>
References: <20161027155336.DBCE7DA329@eggemoggin.niobe.net>
	<4E125A2F-4C11-4F56-92E7-E08E4041DD04@univ-mlv.fr>
Message-ID: <OF3AF87C98.DC71DFD2-ON00258059.0061B87E-86258059.0061F233@notes.na.collabserv.com>

I like this proposal also.  It nicely solves the usecases I have in mind. 
Thanks.

Tom




> From: Remi Forax <forax at univ-mlv.fr>
> To: mark.reinhold at oracle.com, jpms-spec-experts at openjdk.java.net
> Date: 10/27/2016 12:05 PM
> Subject: Re: Proposal (revised^2): 
> #ReflectiveAccessToNonExportedTypes & #AwkwardStrongEncapsulation: 
> Open modules & open packages
> Sent by: "jpms-spec-observers" 
<jpms-spec-observers-bounces at openjdk.java.net>
> 
> I really like this proposal.
> 
> Remi 
> 
> On October 27, 2016 5:53:36 PM GMT+02:00, mark.reinhold at oracle.com 
wrote:
> >Issue summary
> >-------------
> >
> >  #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
> >  libraries require reflective access to members of the non-exported
> > types of other modules; examples include dependency injection (Guice),
> >  persistence (JPA), debugging tools, code-automation tools, and
> >  serialization (XStream).  In some cases the particular library to be
> > used is not known until run time (e.g., Hibernate and EclipseLink both
> > implement JPA).  This capability is also sometimes used to work around
> >  bugs in unchangeable code.  Access to non-exported packages can, at
> >  present, only be done via command-line flags, which is extremely
> >  awkward.  Provide an easier way for reflective code to access such
> >  non-exported types. [1]
> >
> >  #AwkwardStrongEncapsulation --- A non-public element of an exported
> >package can still be accessed via the `AccessibleObject::setAccessible`
> >  method of the core reflection API.  The only way to strongly
> >  encapsulate such an element is to move it to a non-exported package.
> >  This makes it awkward, at best, to encapsulate the internals of a
> >  package that defines a public API. [2]
> >
> >Proposal
> >--------
> >
> >(This is the third proposal for #ReflectiveAccessToNonExportedTypes.
> > The note that follows [3] summarizes the background and relates the
> > alternatives presented thus far.)
> >
> >Extend the language of module declarations with the concept of _open_
> >modules.  Open modules make it easy to define modules whose internals
> >will be accessed at run time by reflection-based frameworks.  Every
> >package in an open module has the following properties, by default:
> >
> >  (1) At compile time it is not exported.
> >
> >  (2) At run time it is exported without regard to split-package
> >      conflicts or other inconsistencies, i.e., it does not affect
> >      the operation of the resolver, nor the construction of layer
> >      configurations, but is simply exported as if by invoking the
> >      `Module::addExports` method during layer instantiation.
> >
> >  (3) At run time it is available for _deep reflection_, i.e., all
> >      of its elements are accessible via the core reflection API
> >      including non-public elements, which can be accessed via the
> >      `AccessibleObject::setAccessible` method.
> >
> >A package in an open module can be exported explicitly, for use at
> >compile time, via the familiar `exports` directive, so property (1)
> >will
> >not hold.  In that case it will also partake in run-time resolution, so
> >property (2) will not hold either.
> >
> >Suppose we have a module `foo.bar` that contains an API package
> >`com.foo.bar`, whose public types are intended for use by other
> >modules,
> >and a non-API package `com.foo.bar.model`, that contains entity classes
> >to be manipulated by Hibernate via core reflection.  Then the module
> >declaration
> >
> >    open module foo.bar {
> >        exports com.foo.bar;
> >        requires hibernate.core;
> >        requires hibernate.entitymanager;
> >    }
> >
> >makes all elements, public and otherwise, of all packages available for
> >deep reflection at run time, but makes only the public and protected
> >types in `com.foo.bar` accessible at compile time.  Thus Hibernate can
> >access non-public elements of the `com.foo.bar.model` package via the
> >`setAccessible` method, and some other module that `requires foo.bar`
> >can be compiled against the API in the `com.foo.bar` package but not
> >against the types in any other package.
> >
> >Open modules can be considered an intermediate step on the migration
> >path, between automatic modules and normal modules:
> >
> >  - An automatic module offers the traditional level of encapsulation:
> >    All packages are both open for deep reflective access and exported
> >   for ordinary compile-time and run-time access to their public types.
> >
> > - An open module offers a modest degree of stronger encapsulation: All
> >  packages are open for deep reflective access, but the module's author
> >    must specify which packages, if any, are exported for ordinary
> >    compile-time and run-time access.
> >
> >  - A normal module offers the strongest encapsulation: The module's
> >    author must specify which packages, if any, are open, or exported,
> >    or both.
> >
> >An open module is a good starting point for application code that's
> >being
> >modularized by its author.  It affords a separation of concerns, so
> >that
> >the author can focus on defining the module's domain-specific API
> >without
> >having to worry about how the module will be inspected or manipulated
> >at
> >run time by reflective frameworks.
> >
> >                                  * * *
> >
> >To open specific packages in normal module declarations we introduce a
> >new per-package directive, `opens`.  A package in a normal module can
> >be
> >opened, or exported, or both, as follows:
> >
> >  - If a package is only opened (`opens p;`) then properties (1), (2),
> >    and (3) hold, i.e., its types are not accessible at compile time,
> >   it does not affect resolution, and all of its elements are available
> >    for deep reflection at run time.
> >
> >  - If a package is only exported (`exports p;`) then those properties
> >    do not hold, and it is exported exactly as it is today.
> >
> >  - If a package is both exported and opened (`exports p; opens p;`)
> >    then it is exported as today and is, additionally, available for
> >    deep reflection at run time (3).
> >
> >We can use the `opens` directive to refine the previous example so that
> >only the `com.foo.bar` package is accessible at compile time, only the
> >`com.foo.bar.model` package is available for deep reflection at run
> >time,
> >and the non-public elements of all other packages are strongly
> >encapsulated:
> >
> >    module foo.bar {
> >        exports com.foo.bar;
> >        opens com.foo.bar.model;
> >        requires hibernate.core;
> >        requires hibernate.entitymanager;
> >    }
> >
> >It's possible to both open and export a package in a normal module, but
> >it's often inadvisable.  This is especially so for API packages, since
> >normally an API's internal implementation details should be strongly
> >encapsulated.  This combination of directives may, however, be useful
> >for
> >legacy APIs whose internals are known to be accessed by existing code.
> >
> >To ensure the integrity of the platform we expect that all the modules
> >of
> >the JDK itself will be normal modules and that very few, if any,
> >packages
> >will be opened.
> >
> >The `opens` directive cannot be used in an `open` module, since all
> >packages in such modules are implicitly open.
> >
> >                                  * * *
> >
> >Both the `exports` and `opens` directives can be qualified, so that a
> >package is exported or opened only to certain other named modules.  As
> >before, duplicate directives are not permitted in order to ensure easy
> >readability.  At most one `exports` directive is relevant to any given
> >package, and at most one `opens` directive is relevant to any given
> >package.
> >
> >There is no syntax for wildcards.  If the defaults are not sufficient
> >for
> >a package then the package must be named explicitly.
> >
> >                                  * * *
> >
> >The existing syntax of `requires public` has long been confusing, so we
> >here take the opportunity to fix that problem by renaming the `public`
> >modifier in `requires` directives to `transitive`.  Thus the
> >declaration
> >
> >    module foo.bar {
> >        exports com.foo.bar;
> >        requires public java.sql;
> >    }
> >
> >is now written
> >
> >    module foo.bar {
> >        exports com.foo.bar;
> >        requires transitive java.sql;
> >    }
> >
> >This is potentially confusing in a different way, since in mathematics
> >the term "transitive" is usually applied to an entire relation rather
> >than to three specific elements of a set.  Its use here does not, in
> >particular, mean that the resolver does not interpret plain `requires`
> >directives when computing the transitive closure of a set of root
> >modules.  "Transitive" as used here is in the more abstract sense,
> >expressing the notion of conveying a property -- in this case, the
> >readability of the required module -- from one thing to another.  On
> >balance, however, `transitive` does appear superior to `public`.
> >
> >Notes
> >-----
> >
> >  - This proposal is significantly different from both the first and
> >    second proposals [4][5].  It recognizes that, in practice, it will
> >    be common to have open modules, all of whose content is available
> >  for deep reflection but otherwise inaccessible at compile time unless
> >  explicitly exported.  If finer-grained control over whether a package
> >    is opened, exported, or both is required then a normal module
> >    declaration can be used.
> >
> >  - This proposal is similar to one made by Stephen Colebourne [6],
> >    in which he uses the term `exposes`, and to various unpublished
> >    proposals (of which there have been many!).  We here adopt the term
> >    `opens` rather than `exposes` in order to avoid confusion between
> >    the phonetically-similar `exports` and `exposes`, especially for
> >    non-native speakers of English.  We also use `open` to modify the
> >    `module` keyword rather than introduce wildcards for the `opens`
> >   directive, since the former is easier to explain.  The latter would,
> >  further, lead people to expect wildcards for the `exports` directive,
> >    but we'd prefer to avoid that since it could encourage the careless
> >    exportation of all of a module's packages.  Exporting a package for
> >    use as an API should always be an explicit, affirmative choice.
> >
> >  - A separate `opens` directive, and the avoidance of wildcards,
> >    simplifies the interactions amongst qualified and unqualified
> >    package-level directives.  We no longer need complex rules such as
> >   those shown for the `private` modifier in the previous proposal [5].
> >
> >- This proposal implies corresponding changes to the `ModuleDescriptor`
> >    and `Module` APIs, but these are reasonably straightforward.
> >
> >  - This proposal amends the proposal for #ResourceEncapsulation [6]
> >    to replace the use of private exports with open modules and open
> >    packages.  Wherever that proposal indicates that a resource can be
> >    located when its effective package name identifies a package that's
> >    exported privately, assume instead that the resource can be located
> >    when that package is either an element of an open module or an open
> >    package in a normal module.
> >
> >  - If a container is to ensure that a package in an application module
> >    is available for deep reflection only by a trusted framework module
> >    then it can arrange for that by rewriting that module's descriptor,
> >    as suggested previously [8], to insert the appropriate qualified
> >  `opens` directive.  This works even in the case, pointed out by Jason
> >    Greene [9], of two modules that open a package of the same name to
> >    the same framework module, since opened and unexported packages are
> >    not subject to the usual package-consistency checks.  There is no
> >    need any longer for the resolution algorithm to take this type of
> >    scenario into account [a].  A more flexible way to arrange for such
> >  precisely-constrained access is a separate issue and may be addressed
> >    by a future proposal.
> >
> >- This proposal primarily addresses "friendly" uses of reflection, such
> >    as dependency injection and persistence, in which the author of a
> >    module knows in advance that one or more, or possibly all, packages
> >   must be opened at run time for deep reflective access by frameworks.
> >  Intrusive access to arbitrary packages of arbitrary modules by, e.g.,
> >    serialization frameworks or debugging tools, will still require the
> >    use of sharp knives such as `--add-exports` or `--add-opens`
> >    command-line options, the legacy unsupported `sun.misc.Unsafe` API
> >    and related APIs, or JVM TI.
> >
> >- Using `--add-exports` or `--add-opens` command-line options, or their
> >  equivalents, remains awkward, but sometimes they're the only way out.
> >   To ease migration it's worth considering some way for an application
> >    packaged as a JAR file to include such options in its `MANIFEST.MF`
> >    file, as suggested by Simon Nash [b].  This is addressed in the
> >    proposal for #AddExportsInManifest [c], which is hereby amended to
> >    rename the `Add-Exports-Private` attribute to `Add-Opens`.
> >
> >
> >[1]
> >http://openjdk.java.net/projects/jigsaw/spec/issues/
> #ReflectiveAccessToNonExportedTypes
> >[2]
> >http://openjdk.java.net/projects/jigsaw/spec/issues/
> #AwkwardStrongEncapsulation
> >[3]
> >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-
> October/000431.html
> >[4]
> >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/
> 000307.html
> >[5]
> >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-
> September/000390.html
> >[6]
> >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-
> September/000392.html
> >[7]
> >
http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html

> >[8]
> >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
> >[9]
> >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
> >[a]
> >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
> >[b]
> >
http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html

> >[c]
> >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-
> September/000391.html
> 
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> 



From mark.reinhold at oracle.com  Thu Oct 27 23:15:37 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Thu, 27 Oct 2016 16:15:37 -0700
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &
	#AwkwardStrongEncapsulation: Weak modules & private exports
In-Reply-To: <279bd57d-515a-a983-ffbb-6f569c29495e@redhat.com>
References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
	<5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com>
	<20161011151514.A04ECD50FA@eggemoggin.niobe.net>
	<279bd57d-515a-a983-ffbb-6f569c29495e@redhat.com>
Message-ID: <20161027161537.655286378eggemoggin.niobe.net>

2016/10/24 6:46:11 -0700, david.lloyd at redhat.com:
> On 10/11/2016 10:15 AM, mark.reinhold at oracle.com wrote:
>> 2016/9/16 8:30:41 -0700, david.lloyd at redhat.com:
>>> ...
>>> 
>>> #1) The "weak" designation appears to be pejorative
>>> 
>>> Under this solution, many existing frameworks and/or the modules which
>>> consume them must be marked "weak" in order to work correctly, or else a
>>> much more complex module descriptor must be used.  While this seems to
>>> be a stepping stone for migration, we find that it rather makes
>>> middleware appear as a second-class citizen.
>> 
>> No insult is intended; "weak" is used here purely in a technical sense,
>> as the obvious antonym of "strong".  Suggestions for a better word for
>> this concept are welcome.
> 
> It's the concept itself that is problematic, I think.  Really, if I 
> understand correctly, the "weak" notion exists merely as shorthand to 
> say "grant permission on all packages to all modules for private 
> reflection", which is the sort of thing that a user ought to be saying 
> explicitly.  Many users will use weak modules without understanding what 
> they are; some will come to understand it and immediately press for best 
> practices to get rid of them.  I think that regardless of what the 
> concept is named, it's going to spell trouble in the end.

Many others had similar reactions, for a variety of reasons, so
please have a look at the open modules & open packages proposal
I posted earlier today [1].

>>> ...
>>> 
>>> #3) Controlling reflection access is always bound up with controlling
>>> exports
>>> 
>>> The proposed exports system is simplistic but not exactly simple, yet
>>> several useful modes remain left unaccounted-for.  Internally we
>>> examined the usefulness of controlling access to public and so-called
>>> "deep" reflection in combination with whether or not a member was
>>> exported, and we came up with this logical table:
>>> 
>>>          +------------------+---------------------------+
>>>          |                  |        Module is...       |
>>>          |  Reflection is... +------------+--------------+
>>>          |                  |  Exported  | Not Exported |
>>>          +----------------------------------------------+
>>>          |        Forbidden | Not Useful |    Useful    |
>>>          +----------------------------------------------+
>>>          |      Public Only |   Useful   |    Useful    |
>>>          +----------------------------------------------+
>>>          | Public & Private |   Useful   |    Useful    |
>>>          +------------------+------------+--------------+
>> 
>> I'm not sure how to read this table.  Should the cell containing the text
>> "Module is ..." instead contain "Member is ..."?
> 
> Right.
> 
>> By "exported" I'll assume that you mean non-reflective access, i.e., from
>> the language at compile time and bytecodes at run time.
> 
> Yes.
> 
>> Do you have example use cases for each of the "Useful" cells?  Some are
>> obvious, but others are not.
> 
> Sure, I'll go over each one by one with some example use cases.
> 
> Forbidden/Not Exported: Obviously the foundation case for 
> access-oriented encapsulation, this is the default mode.
> 
> Exported modes:
> 
> Public Only/Exported: It doesn't make sense to disallow reflection on 
> public types and members that can already be accessed directly, so this 
> would be the default access mode to imported members when one module 
> imports another.
> 
> Public & Private/Exported: This is the JDK 8 compatibility mode, but a 
> module should have to opt into this mode, and it should be restricted to 
> specific targets whenever possible, else the mechanism is essentially 
> defeated.
> 
> Not exported modes:
> 
> Public Only/Not Exported: This is the mode that would be used (for 
> example) by DI and ORM frameworks and containers which operate on public 
> members.  Normally a module which imports such a framework would grant 
> the necessary reflective access to it as well (though a full export is 
> in this case unnecessary); the framework wouldn't contain requires 
> clauses for modules which consume it.  Like I suggested in my first 
> email, a way to "handshake" this agreement seems like it would be 
> necessary: the framework would not allow itself to be imported without 
> the importing module acknowledging the access in its configuration.
> 
> Public & Private/Not Exported: This is the mode that would be used by 
> serialization and DI/ORM frameworks and containers which operate on 
> private members (e.g. field-level injection).  Like the above though, 
> this level of access is usually granted by the module importing the 
> framework in question and should be explicitly opted into if possible.

Now that I understand it, I think your table maps fairly directly to the
analysis that's driven my current and past proposals, which I summarized
earlier today [2].  Please let me know if you think otherwise.

The open-modules/packages proposal does not support your "Public Only /
Not Exported" mode, which corresponds to the missing "N S" row in the
compact table in [2], i.e., the "partly-encapsulated framework client"
case.  Most reflective frameworks in use today will just reach in via
`setAccessible` whenever they need to, so whether the members of an
unexported package in a client module are public or private doesn't
really matter.  The new proposal intentionally does not support this
case, which results in a simpler design.

> The better situation however is to separate the grant from the export.

That's exactly what the new `opens` directive does [1].

>>> #4) There isn't always a container
>>> 
>>> The justification for not providing an easy mechanism to restrict
>>> private reflective access to a trusted framework is that a container can
>>> always rewrite a module or inject classes.  However there isn't always a
>>> container present - something which the existence of weak modules
>>> acknowledges.  Some frameworks and middleware are meant to be usable
>>> with or without a container, and it is undesirable to force these use
>>> cases into a weak module situation or to require a strong coupling in
>>> the module descriptor in this case.  Some other solution should be found
>>> that works both for containers and "regular" applications.
>> 
>> If you're running outside of a container then how important is it to
>> precisely restrict access to packages meant for deep reflection?  If
>> you're outside of a container then you have to trust the person (you?)
>> who sets up your module path and/or class path.  To take the canonical
>> case of a module that just contains JPA model classes, if you make it a
>> weak module or else `exports private` the relevant packages without
>> qualification so that the hibernate.core (or whatever) module can reflect
>> into it, then the scope for trouble is limited.  Sure, some other module
>> on the module path or even code on the class path could reflect into your
>> module when it's not supposed to, but is it worth complicating the module
>> system just to handle this scenario?
> 
> In either situation, the person (deployer) who sets up the class path or 
> installation can currently do whatever they want.  Nothing in the new 
> (or existing) security framework truly makes it possible to run 
> untrusted code safely in this regard, nor to safely run trusted code in 
> an untrusted environment, nor is it a sufficient solution for secure 
> sandboxing even with a security manager present, with or without a 
> container.  At best it's a mitigation strategy.

Exactly.

> That said: the rise of microservices and reactive programming suggests 
> that such standalone (containerless) deployments are, if anything, more 
> popular than ever, but they may suffer from security problems which are 
> very similar to, if not identical to, those which applications running 
> in containers are subject to.

That could be, and it deserves further thought.

>                                In fact jlink (and the new AOT work for 
> that matter) seem particularly and specifically suited to this type of 
> use case.

Agreed.

> Finally: a JPA model class module is indeed probably harmless in any 
> practical security sense, and locking it down is probably not useful 
> from a security perspective, but even in this case it may yet be useful 
> to restrict package exports such that only _supported_ classes are 
> visible (especially when a module is published within an organization 
> and contracts must be maintained); i.e. encapsulation is not solely a 
> security concern, it also establishes intent as clearly as any interface 
> declaration does.

I completely agree that encapsulation is not just for security -- it's
also about helping developers build reliable systems by defining clear
boundaries between elements at all levels (classes, package, modules)
so that systems are understandable, testable, and so forth.

The open-modules/packages proposal does address your last point: In
contrast to the weak-modules proposal, you can prevent some or all of
the packages of a framework-client module from being compiled against,
i.e., treated as API, by opening them but not exporting them.

>>> ...
>>> 
>>> #7) More loose coupling seems necessary and useful
>>> 
>>> In order for typical applications to function with modern middleware as
>>> modules, without compromising security, it may be necessary to enhance
>>> the loose coupling mechanism (uses/provides), or to provide an
>>> additional, similar mechanism, which allows a symbolic coupling which
>>> would allow modules to declare (in an abstract manner) modules which
>>> need to have reflective access to it in a "friendly" manner.  ...
>> 
>> To confirm my understanding, would it be correct to say that your main
>> concern here is scenarios in which some of the classes in a user module
>> are intended to be manipulated reflectively by some trusted framework
>> module, but that framework module is either not known at compile time or
>> is otherwise inconvenient to name in the declaration of the user module?
> 
> Correct.

Good.  I'll enter this as a new issue:

  #IndirectQualifiedReflectiveAccess --- Provide a means by which a
  client module can grant qualified reflective access to a framework
  module that is not known at compile time, assuming that the name of
  some other module that represents the framework module is known.
  A canonical example of this case is a client POJO module compiled
  against a module that defines the JPA API.  At run time the client
  module, or a container or some other code acting on its behalf,
  must grant qualified reflective access to its POJO packages to the
  JPA implementation in actual use.

- Mark


[1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html
[2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html

From forax at univ-mlv.fr  Fri Oct 28 15:35:35 2016
From: forax at univ-mlv.fr (Remi Forax)
Date: Fri, 28 Oct 2016 17:35:35 +0200 (CEST)
Subject: Please welcome Neil Bartlett to JSR 376
In-Reply-To: <0d21c865-414d-0593-f8c1-0c51bdd416f6@redhat.com>
References: <20161025140849.350853791eggemoggin.niobe.net>
	<0d21c865-414d-0593-f8c1-0c51bdd416f6@redhat.com>
Message-ID: <1025055751.2163401.1477668935510.JavaMail.zimbra@u-pem.fr>

Welcome Neil :)

R?mi

----- Mail original -----
> De: "David M. Lloyd" <david.lloyd at redhat.com>
> ?: jpms-spec-experts at openjdk.java.net
> Envoy?: Mardi 25 Octobre 2016 23:49:30
> Objet: Re: Please welcome Neil Bartlett to JSR 376

> On 10/25/2016 04:08 PM, mark.reinhold at oracle.com wrote:
>> Neil Bartlett, a recognized expert on OSGi, has asked to join this EG.
>>
>> Ordinarily I'm extremely reluctant to accept new EG members this late in
>> the life of a JSR, long after we've settled the requirements and after
>> we've been working on a design for over a year.  Peter Kriens's recent
>> departure, however, leaves the EG with no resident OSGi expertise, and
>> that's an important gap to fill.
>>
>> Neil has agreed to accept the requirements as they stand and, further,
>> acknowledges that our primary goal at this stage is to finish the design
>> that we have rather than come up with something significantly different.
>> I'm confident that he'll bring a valuable perspective to our discussions
>> and help to ensure the overall success of the JSR.
>>
>> (Neil -- if there's anything further you'd like to say by way of
>>  introduction, please do so!)
> 
> Welcome aboard, Neil!
> --
> - DML

From forax at univ-mlv.fr  Fri Oct 28 15:41:41 2016
From: forax at univ-mlv.fr (Remi Forax)
Date: Fri, 28 Oct 2016 17:41:41 +0200 (CEST)
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <20161026102447.63677664eggemoggin.niobe.net>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
	<20161026091545.152155830eggemoggin.niobe.net>
	<OF73A10E29.8203919F-ON00258058.0059BBC5-86258058.005AE60E@notes.na.collabserv.com>
	<20161026102447.63677664eggemoggin.niobe.net>
Message-ID: <960310567.2164815.1477669301699.JavaMail.zimbra@u-pem.fr>



----- Mail original -----
> De: "mark reinhold" <mark.reinhold at oracle.com>
> ?: "Thomas Watson" <tjwatson at us.ibm.com>
> Cc: jpms-spec-experts at openjdk.java.net
> Envoy?: Mercredi 26 Octobre 2016 19:24:47
> Objet: Re: Dynamic modules and bidirectional interoperation

> 2016/10/26 9:32:42 -0700, Thomas Watson <tjwatson at us.ibm.com>,
> jpms-spec-comments:
>> 2016/10/26 09:15:45 -0700, mark.reinhold at oracle.com, jpms-spec-experts:
>>> 2016/10/7 8:47:53 -0700, Thomas Watson <tjwatson at us.ibm.com>,
>>> jpms-spec-observers:
>>>> ...
>>>> 
>>>> 1) Any OSGi dependencies that influence class loader delegation will not
>>>> be declared in the ModuleDescriptors which represent the OSGi bundle
>>>> wirings.  Instead we will depend on the ability to
>>>> #ReadabilityAddedByLayerCreator to wire up the modules within the bundle
>>>> Layer according to OSGi resolution rules.  It has a drawback if there are
>>>> cases where some system code is trying to reason about modular
>>>> dependencies by using the JPMS module API.  I think this will have to be
>>>> acceptable because I don't think we will be able to fully converge on all
>>>> the possible dependency types available in the existing module systems.
>>>> For example, OSGi has Require-Bundle which maps closing to JPMS requires,
>>>> but OSGi also has Import-Package dependencies which cannot be mapped
>>>> properly into JMPS module Layers.
>>> 
>>> If by "system code" you mean code in the JDK itself, I don't think this
>>> will be a big problem at run time.  Except for diagnostic purposes,
>>> module descriptors are really only interpreted during resolution and
>>> configuration.  All other run-time reasoning about module readability
>>> is based upon the reads edges in the module graph as reported by the
>>> `java.lang.reflect.Module::canRead` method, and that will include any
>>> reads edges that you add to support bundle wirings.
>> 
>> Here I am using "system code" in the general sense, but I think any
>> "system code" that cares about wiring will need to pay attention to
>> Module::canRead like you suggest.  I can think of a few frameworks that
>> care about how modules are wired up at runtime.  For example, in OSGi we
>> have the extender pattern. Without going into much detail here, basically
>> an extender implementation module will only act on extendee modules which
>> are wired to the extender module.  If such a pattern exists using JPMS
>> then the "system code" would be the extender implementation.
> 
> Okay, I see.
> 
>>>> 2) I assume #NonHierarchicalLayers will be restricted to form a directed
>>>> acyclic graph.
>>> 
>>> Yes.
>>> 
>>>>                 As such I assume each node in the graph of Layers would be
>>>> visited only once when resolving new Layers on top even if there are
>>>> multiple paths to the layer from the new Layer being resolved.
>>> 
>>> Yes.
>> 
>> The motivation for my question was to ask some follow-ons.
>> 
>> Will there will be a specified order in which the nodes are traversed
>> when there are multiple paths to a node?  More specifically will the
>> parents have a specified order in which they are delegated to?
> 
> Yes, the parents of a new configuration will be specified in an ordered
> list and during resolution they'll be searched depth-first, in the order
> in which they occur in the list.  Depth-first order makes it easy to
> reason about resolution, since it works just like a search path.

I fail to seen how this is not a mini-classpath with exactly the same issues as the classpath.

> 
> - Mark

R?mi

From forax at univ-mlv.fr  Fri Oct 28 16:01:31 2016
From: forax at univ-mlv.fr (forax at univ-mlv.fr)
Date: Fri, 28 Oct 2016 18:01:31 +0200 (CEST)
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <OF28202619.DFE819DF-ON00258056.0064110A-86258056.006C0D8F@notes.na.collabserv.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr>
	<OF28202619.DFE819DF-ON00258056.0064110A-86258056.006C0D8F@notes.na.collabserv.com>
Message-ID: <531946364.2170171.1477670491677.JavaMail.zimbra@u-pem.fr>

Hi Thomas, 
first, sorry, 
I'm not able to parse my own mail correctly. 

Let's try again, 
currently a way to solve your issue is to allow a Layer to have several parents, 
i really dislike this idea because it introduces an arbitrary order that is hidden deep in the code thus makes really hard to reason about which Layer resolve what, 
for me it's not different to add a kind of classpath resolution in the middle of the module resolution. 

Currently, you control the classloaders, so you can load a hand crafted class in the module (you just have to patch the class name to have the right package, you can use the patching capabilities of defineAnonymousClass for that, but you can also call defineClass on the right classloader as again, you control the classloaders). 
Once you are able to execute arbitrary code in a module you can call addReads, addExports (and soon addOpens) on the j.l.r.Module. 

regards, 
R?mi 

> De: "Thomas Watson" <tjwatson at us.ibm.com>
> ?: forax at univ-mlv.fr, "mark reinhold" <mark.reinhold at oracle.com>,
> "jpms-spec-observers" <jpms-spec-observers at openjdk.java.net>
> Envoy?: Lundi 24 Octobre 2016 21:40:06
> Objet: Re: Dynamic modules and bidirectional interoperation

> Can you explain more about how monkey patching would be done in a JVM
> implementation neutral way? If I understand correctly your suggestion would
> require monkey patching in order to add functionality to the module classes
> provided by the boot layer (Module, Layer, Configuration etc.). This additional
> functionality would allow an OSGi Framework implementation to represent the
> dynamic set of bundle wirings in an OSGi Framework as JPMS modules.

> In order to do that it seems we would need extensive knowledge of the internal
> implementation details of the classes we are monkey patching in order to modify
> their internal structures. I can see how using Unsafe would give us the power
> to hack into the implementation details of jigsaw, but I don't view that as a
> satisfactory solution to the problem. It would bind us to a particular internal
> implementation of Java 9 in order to work correctly. It places a new dependency
> on Unsafe class which is from the jdk.unsupported module. I don't think new
> dependencies on the jdk.unsupported module should be advocated. Doing so has a
> high probability of breaking on a future version of Java if Unsafe is removed
> by default or the implementation details of the module classes change.

> Tom

> "jpms-spec-observers" <jpms-spec-observers-bounces at openjdk.java.net> wrote on
> 10/21/2016 08:24:38 AM:

> > From: Remi Forax <forax at univ-mlv.fr>
> > To: mark reinhold <mark.reinhold at oracle.com>
> > Cc: jpms-spec-experts at openjdk.java.net
> > Date: 10/21/2016 08:25 AM
> > Subject: Re: Dynamic modules and bidirectional interoperation
> > Sent by: "jpms-spec-observers" <jpms-spec-observers-bounces at openjdk.java.net>

> > Hi Mark,
> > I see an other way to solve the wiring issue,
> > use VM anonymous classes (via unsafe.defineAnonymousClass) to
> > generate the glue code (as lambda meta factory does to implement the
> > functional interface at runtime) and method handles as wires (once
> > created a method handle do not do any security check anymore, so the
> > requirement to be in a module with specific parents can be lifted).
> > With that, you don't need to generate the wiring code inside a new
> > module and when a new bundle need to be re-wire, you can mutate the
> > wire (the method handle) by using a mutable callsite.

> > Said differently, loading/unloading an OSGI bundle is equivalent to
> > doing monkey patching in a dynamic language, so there is no reason
> > to not use the tools we provide for dynamic languages to wire OSGI bundles.

> > so i vote strongly against that proposal mostly because it's too
> > late to change requirements and because some obvious design options
> > to implement OSGI on top of JPMS have not been tested.

> > regards,
> > R?mi

> > ----- Mail original -----
> > > De: "mark reinhold" <mark.reinhold at oracle.com>
> > > ?: jpms-spec-experts at openjdk.java.net
> > > Envoy?: Jeudi 6 Octobre 2016 19:07:59
> > > Objet: Dynamic modules and bidirectional interoperation

> > > The agreed requirements for this JSR mandate a modest degree of dynamic
> > > configuration [1]. They do not mandate _dynamic modules_, i.e., the
> > > fine-grained dynamic configuration or "hot deployment" that is possible
> > > with other module systems, such as OSGi and JBoss, in which individual
> > > modules can be installed, updated, and uninstalled at any time.

> > > The requirements also mandate that it "be possible for another module
> > > system, such as OSGi, to locate Java modules and resolve them using its
> > > own resolver, except possibly for core system modules" [2]. They do not
> > > mandate _bidirectional interoperation_, which in the case of OSGi would
> > > be the ability to arrange for bundles to refer to JPMS modules and also
> > > vice versa, i.e., for JPMS modules to refer to bundles.


From mark.reinhold at oracle.com  Fri Oct 28 16:16:49 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Fri, 28 Oct 2016 09:16:49 -0700
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <960310567.2164815.1477669301699.JavaMail.zimbra@u-pem.fr>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
	<20161026091545.152155830eggemoggin.niobe.net>
	<20161026102447.63677664eggemoggin.niobe.net>
	<960310567.2164815.1477669301699.JavaMail.zimbra@u-pem.fr>
Message-ID: <20161028091649.617098323eggemoggin.niobe.net>

2016/10/28 8:41:41 -0700, Remi Forax <forax at univ-mlv.fr>:
> De: "mark reinhold" <mark.reinhold at oracle.com>
> Envoy?: Mercredi 26 Octobre 2016 19:24:47
>> 2016/10/26 9:32:42 -0700, Thomas Watson <tjwatson at us.ibm.com>,
>> jpms-spec-comments:
>>> ...
>>> 
>>> Will there will be a specified order in which the nodes are traversed
>>> when there are multiple paths to a node?  More specifically will the
>>> parents have a specified order in which they are delegated to?
>> 
>> Yes, the parents of a new configuration will be specified in an ordered
>> list and during resolution they'll be searched depth-first, in the order
>> in which they occur in the list.  Depth-first order makes it easy to
>> reason about resolution, since it works just like a search path.
> 
> I fail to seen how this is not a mini-classpath with exactly the same
> issues as the classpath.

It's not a "mini" class path; it's more like a "mini" or "run time"
module path.  It's better than a class path because it's a means to
locate whole modules rather than individual types, as discussed in
SotMS:

  http://openjdk.java.net/projects/jigsaw/spec/sotms/#the-module-path

- Mark

From forax at univ-mlv.fr  Fri Oct 28 17:01:17 2016
From: forax at univ-mlv.fr (forax at univ-mlv.fr)
Date: Fri, 28 Oct 2016 19:01:17 +0200 (CEST)
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <20161028091649.617098323eggemoggin.niobe.net>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
	<20161026091545.152155830eggemoggin.niobe.net>
	<20161026102447.63677664eggemoggin.niobe.net>
	<960310567.2164815.1477669301699.JavaMail.zimbra@u-pem.fr>
	<20161028091649.617098323eggemoggin.niobe.net>
Message-ID: <507939175.2182737.1477674077829.JavaMail.zimbra@u-pem.fr>



----- Mail original -----
> De: "mark reinhold" <mark.reinhold at oracle.com>
> ?: "Remi Forax" <forax at univ-mlv.fr>
> Cc: jpms-spec-experts at openjdk.java.net
> Envoy?: Vendredi 28 Octobre 2016 18:16:49
> Objet: Re: Dynamic modules and bidirectional interoperation

> 2016/10/28 8:41:41 -0700, Remi Forax <forax at univ-mlv.fr>:
>> De: "mark reinhold" <mark.reinhold at oracle.com>
>> Envoy?: Mercredi 26 Octobre 2016 19:24:47
>>> 2016/10/26 9:32:42 -0700, Thomas Watson <tjwatson at us.ibm.com>,
>>> jpms-spec-comments:
>>>> ...
>>>> 
>>>> Will there will be a specified order in which the nodes are traversed
>>>> when there are multiple paths to a node?  More specifically will the
>>>> parents have a specified order in which they are delegated to?
>>> 
>>> Yes, the parents of a new configuration will be specified in an ordered
>>> list and during resolution they'll be searched depth-first, in the order
>>> in which they occur in the list.  Depth-first order makes it easy to
>>> reason about resolution, since it works just like a search path.
>> 
>> I fail to seen how this is not a mini-classpath with exactly the same
>> issues as the classpath.
> 
> It's not a "mini" class path; it's more like a "mini" or "run time"
> module path.  It's better than a class path because it's a means to
> locate whole modules rather than individual types, as discussed in
> SotMS:
> 
>  http://openjdk.java.net/projects/jigsaw/spec/sotms/#the-module-path

it's not a modulepath it's a moduletreepath,
so i agree it does not have exactly same issue as the classpath, but the main issue is very similar, if you take a look to the list of parents, it's not enough to see which module will be chosen, like with the classpath, if you open all the jars you will find your cookie, with the moduletreepath, if you crawle the tree in deep-first order, you will find your module, in my opinion, in both cases, if you take a look to the list of jars or the list of parents,
you have no idea where to find what you're looking for. And here, it's even worst, because at least the classpath has a string representation.

> 
> - Mark

R?mi

From tjwatson at us.ibm.com  Fri Oct 28 19:20:51 2016
From: tjwatson at us.ibm.com (Thomas Watson)
Date: Fri, 28 Oct 2016 14:20:51 -0500
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <531946364.2170171.1477670491677.JavaMail.zimbra@u-pem.fr>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr>
	<OF28202619.DFE819DF-ON00258056.0064110A-86258056.006C0D8F@notes.na.collabserv.com>
	<531946364.2170171.1477670491677.JavaMail.zimbra@u-pem.fr>
Message-ID: <OF3B857C8B.7893E486-ON0025805A.00633750-8625805A.006A4B1B@notes.na.collabserv.com>

> From: forax at univ-mlv.fr
> To: Thomas Watson/Austin/IBM at IBMUS
> Cc: mark reinhold <mark.reinhold at oracle.com>, jpms-spec-observers 
> <jpms-spec-observers at openjdk.java.net>
> Date: 10/28/2016 11:01 AM
> Subject: Re: Dynamic modules and bidirectional interoperation
> 
> Hi Thomas,
> first, sorry,
> I'm not able to parse my own mail correctly.
> 
> Let's try again,
> currently a way to solve your issue is to allow a Layer to have 
> several parents,
> i really dislike this idea because it introduces an arbitrary order 
> that is hidden deep in the code thus makes really hard to reason 
> about which Layer resolve what,
> for me it's not different to add a kind of classpath resolution in 
> the middle of the module resolution.

I don't have near the history with JPMS as this group. I'm going out on a 
limb talking about the definitive behavior of the JMPS runtime, but here 
goes. Today layers are searched for required modules in a specific single 
parent order.  First the local layer is searched, if not found look in the 
parent layer, do this until the empty layer is reached.  For example, look 
at the following layers:

 A -> B -> C 

These are simply searched in order A, B, C.
Now we introduce multiple parents:

 A -> B -> C
  \-> D -> E

These are searched in order A, B, C, D, E.  From the perspective of layer 
A I would expect a consistent resolution result as if you had a single 
parent hierarchy.

 A -> B -> C -> D -> E

The problem I do see though is what happens if you have the following:

 layer B has modules X and Z
 layer D has modules Y and Z 
 both modules X and Y requires public Z
 layer A has module W which requires both X and Y

This better result in some kind of error regardless if single or 
multi-parent layers are used because in both cases module W is getting 
exposed to two copies of the Z module (through dependencies X.B->Z.B and 
Y.D->Z.D).  I'm basing this on the javadoc of Configuration for 
findModule(String name) and resolveRequires(ModuleFinder before, 
ModuleFinder after, Collection<String> roots).  In my opinion Z.B is the 
only Z module which module W should ever be able to wire to because if it 
requires Z directly it could only ever wire to Z.B.  But this may be 
over-restrictive. For example, what if W only requires Y? In that case it 
seems perfectly reasonable to allow W to wire to Y.D and Z.D. Is this 
problem made worse with multiple parent layers?

This approach of using #NonHierarchicalLayers is suggested as a tool to 
solve the problem of pinning modules that are dynamically updated or 
uninstalled in a dynamic module system like OSGi.  Another possible 
solution to that is to use a solution to #DiscardableModules, but that 
does require an ever growing chain of parent Layers to be created as 
modules are updated in the dynamic module system.

> 
> Currently, you control the classloaders, so you can load a hand 
> crafted class in the module (you just have to patch the class name 
> to have the right package, you can use the patching capabilities of 
> defineAnonymousClass for that, but you can also call defineClass on 
> the right classloader as again, you control the classloaders).
> Once you are able to execute arbitrary code in a module you can call
> addReads, addExports (and soon addOpens) on the j.l.r.Module.

I agree with you this could be used instead of a solution to 
#ReadabilityAddedByLayerCreator, but I think it is a bit messy when it 
doesn't have to be.  A solution to #ReadabilityAddedByLayerCreator should 
not be too controversial. I also think this is an orthogonal issue to the 
pinning stale modules issue. I don't think you are suggesting this 
approach to somehow reach into the Layer implementation to pluck out the 
stale modules ourselves. That is where the monkey-patching feels more like 
gorilla-patching to me.

> 
> regards,
> R?mi
> 

Tom



From mark.reinhold at oracle.com  Fri Oct 28 22:48:29 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Fri, 28 Oct 2016 15:48:29 -0700
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <507939175.2182737.1477674077829.JavaMail.zimbra@u-pem.fr>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<OFBE4DEE2C.E182DEFC-ON00258044.0075228B-86258045.0056C9C9@notes.na.collabserv.com>
	<20161026091545.152155830eggemoggin.niobe.net>
	<20161028091649.617098323eggemoggin.niobe.net>
	<507939175.2182737.1477674077829.JavaMail.zimbra@u-pem.fr>
Message-ID: <20161028154829.70438218eggemoggin.niobe.net>

2016/10/28 10:01:17 -0700, Remi Forax <forax at univ-mlv.fr>:
>> De: mark.reinhold at oracle.com
>> Envoy?: Vendredi 28 Octobre 2016 18:16:49
>> 2016/10/28 8:41:41 -0700, Remi Forax <forax at univ-mlv.fr>:
>>> De: mark.reinhold at oracle.com
>>> Envoy?: Mercredi 26 Octobre 2016 19:24:47
>>>> ...
>>>> 
>>>> Yes, the parents of a new configuration will be specified in an ordered
>>>> list and during resolution they'll be searched depth-first, in the order
>>>> in which they occur in the list.  Depth-first order makes it easy to
>>>> reason about resolution, since it works just like a search path.
>>> 
>>> I fail to seen how this is not a mini-classpath with exactly the same
>>> issues as the classpath.
>> 
>> It's not a "mini" class path; it's more like a "mini" or "run time"
>> module path.  It's better than a class path because it's a means to
>> locate whole modules rather than individual types, as discussed in
>> SotMS:
>> 
>> http://openjdk.java.net/projects/jigsaw/spec/sotms/#the-module-path
> 
> it's not a modulepath it's a moduletreepath, so i agree it does not
> have exactly same issue as the classpath, but the main issue is very
> similar, if you take a look to the list of parents, it's not enough to
> see which module will be chosen, like with the classpath, if you open
> all the jars you will find your cookie, with the moduletreepath, if
> you crawle the tree in deep-first order, you will find your module, in
> my opinion, in both cases, if you take a look to the list of jars or
> the list of parents, you have no idea where to find what you're
> looking for.

Ah, now I think I see what you're getting at, which is a completely
different issue.

Yes, it's true that all of the parents, direct and indirect, of a new
configuration are logically arranged in a tree rather than a linear
path, owing to the depth-first search.  This is a behavior shared with
the class path (cf. the `Class-Path` attribute in `MANIFEST.MF`).

As long as we specify it clearly then I'm not convinced that the
complexity is a serious problem, since only a few experts are likely
ever to use this API.

>              And here, it's even worst, because at least the classpath
> has a string representation.

Why is that interesting?  These lists of parent configurations (and any
related lists of layers) will only ever be specified programmatically.
We're not proposing this for use on the command line, or in system
properties, or in any other context that needs a string representation.

- Mark

From mark.reinhold at oracle.com  Mon Oct 31 18:31:57 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Mon, 31 Oct 2016 11:31:57 -0700
Subject: Dynamic modules and bidirectional interoperation
In-Reply-To: <OF3B857C8B.7893E486-ON0025805A.00633750-8625805A.006A4B1B@notes.na.collabserv.com>
References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net>
	<1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr>
	<OF28202619.DFE819DF-ON00258056.0064110A-86258056.006C0D8F@notes.na.collabserv.com>
	<531946364.2170171.1477670491677.JavaMail.zimbra@u-pem.fr>
	<OF3B857C8B.7893E486-ON0025805A.00633750-8625805A.006A4B1B@notes.na.collabserv.com>
Message-ID: <20161031113157.642562730eggemoggin.niobe.net>

2016/10/28 12:20:51 -0700, Thomas Watson <tjwatson at us.ibm.com>:
> ...
> 
> I don't have near the history with JPMS as this group. I'm going out on a 
> limb talking about the definitive behavior of the JMPS runtime, but here 
> goes. Today layers are searched for required modules in a specific single 
> parent order.  First the local layer is searched, if not found look in the 
> parent layer, do this until the empty layer is reached.

This isn't quite right ...

>                                                          For example, look 
> at the following layers:
> 
>  A -> B -> C 
> 
> These are simply searched in order A, B, C.
> Now we introduce multiple parents:
> 
>  A -> B -> C
>   \-> D -> E
> 
> These are searched in order A, B, C, D, E.  From the perspective of layer 
> A I would expect a consistent resolution result as if you had a single 
> parent hierarchy.
> 
>  A -> B -> C -> D -> E

Yes, that's what I expect also.

> The problem I do see though is what happens if you have the following:
> 
>  layer B has modules X and Z
>  layer D has modules Y and Z 
>  both modules X and Y requires public Z
>  layer A has module W which requires both X and Y

(Reminder: `requires public` is being renamed to `requires transitive`.)

Graphically:

  A { W requires X & Y }
  -> B { X requires transitive Z, Z }
  -> D { Y requires transitive Z, Z }

> This better result in some kind of error regardless if single or 
> multi-parent layers are used because in both cases module W is getting 
> exposed to two copies of the Z module (through dependencies X.B->Z.B and 
> Y.D-> Z.D).  I'm basing this on the javadoc of Configuration for 
> findModule(String name) and resolveRequires(ModuleFinder before, 
> ModuleFinder after, Collection<String> roots).

Whether or not this results in an error depends upon how you configure
layer B.

To compute a configuration the resolver takes two module finders, a
"before" finder and an "after" finder.  To locate a module by name it
consults the "before" finder, the parent configuration (and then its
parents recursively), and the "after" finder, in that order, until it
locates a module or fails.

If Z.B is in B's "after" finder (the common case) then no error will be
reported.  Module X in layer B will read the copy of Z in layer D (X.B ->
Z.D), i.e., there will be no Z.B in the resulting module graph.  Even if
you ask for Z explicitly when configuring B, i.e., specify it as a root,
you'll get Z.D rather than Z.D.

If Z.B is in B's "before" finder (the uncommon case, principally to
support `--upgrade-module-path`) then an error will be reported since W
would have to read both Z.B and Z.D due to the read edges implied by the
`requires transitive Z` directives in the definitions of modules X and Y.

(Corner case: If the two Z modules do not define any packages of the same
 name then right now no error is reported; this is probably a bug.)

>                                                 In my opinion Z.B is the 
> only Z module which module W should ever be able to wire to because if it 
> requires Z directly it could only ever wire to Z.B.

No, if W required Z directly then it would still get Z.D since there
wouldn't be a Z.B in B's configuration.

>                                                      But this may be 
> over-restrictive. For example, what if W only requires Y? In that case it 
> seems perfectly reasonable to allow W to wire to Y.D and Z.D.

If W only requires Y then you'd still get W -> Y.D, Z.D.

>                                                               Is this 
> problem made worse with multiple parent layers?

The semantics seem clear enough; whether they're "worse" depends upon
whether you consider the current behavior to be a problem.  The error
scenario described above can now occur between two modules of the same
name in different parents, so in theory at least there would be more
opportunities for this kind of situation to arise.  Whether that would
turn out to be true in practice, I'm not sure.

> This approach of using #NonHierarchicalLayers is suggested as a tool to 
> solve the problem of pinning modules that are dynamically updated or 
> uninstalled in a dynamic module system like OSGi.  Another possible 
> solution to that is to use a solution to #DiscardableModules, but that 
> does require an ever growing chain of parent Layers to be created as 
> modules are updated in the dynamic module system.

Right.

> 2016/10/21 6:24:38 -0700, Remi Forax <forax at univ-mlv.fr>,
> jpms-spec-observers:
>> ...
>> 
>> Let's try again, currently a way to solve your issue is to allow a
>> Layer to have several parents, i really dislike this idea because it
>> introduces an arbitrary order that is hidden deep in the code thus
>> makes really hard to reason about which Layer resolve what, for me
>> it's not different to add a kind of classpath resolution in the
>> middle of the module resolution.
>> 
>> Currently, you control the classloaders, so you can load a hand
>> crafted class in the module (you just have to patch the class name
>> to have the right package, you can use the patching capabilities of
>> defineAnonymousClass for that, but you can also call defineClass on
>> the right classloader as again, you control the classloaders).  Once
>> you are able to execute arbitrary code in a module you can call
>> addReads, addExports (and soon addOpens) on the j.l.r.Module.
> 
> I agree with you this could be used instead of a solution to 
> #ReadabilityAddedByLayerCreator, but I think it is a bit messy when it 
> doesn't have to be.  A solution to #ReadabilityAddedByLayerCreator should 
> not be too controversial. I also think this is an orthogonal issue to the 
> pinning stale modules issue. I don't think you are suggesting this 
> approach to somehow reach into the Layer implementation to pluck out the 
> stale modules ourselves. That is where the monkey-patching feels more like 
> gorilla-patching to me.

Agreed, though I'm not sure that's what R?mi is trying to suggest.

Perhaps a more pertinent problem with R?mi's general idea is that it
would only modify the run-time module graph, as instantiated via layers
and updated via the j.l.r.Module API.  Such updates would have no effect
on the later resolution of JPMS modules since resolution refers to the
unchanging configurations of the relevant layers rather than to the
run-time module graph.  Depending on the details I suspect this would
make it difficult, if not impossible, to resolve JPMS modules against
OSGi bundles surfaced as JPMS modules in the way that I suggested.

- Mark

From mark.reinhold at oracle.com  Mon Oct 31 20:22:38 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Mon, 31 Oct 2016 13:22:38 -0700 (PDT)
Subject: Proposal: #NonHierarchicalLayers
Message-ID: <20161031202238.D3E5DDB037@eggemoggin.niobe.net>

Issue summary
-------------

  #NonHierarchicalLayers --- Layers are presently constrained to be
  hierarchical, i.e., each layer has at most one parent.  Should this
  restriction be relaxed so that a layer can have more than one parent?
  Some have argued that this will be essential to the adoption of the
  module system by a future version of the Java EE Platform.  It would
  also enable bidirectional interoperation with existing module systems
  such as OSGi. [1]


Proposal
--------

Revise `java.lang.module.Configuration` and `java.lang.reflect.Layer` to
augment the existing instance methods that treat `this` as the parent of
the child being created with corresponding static methods that take a
list of parents.

The list of the parents of a configuration can be thought of as a search
path.  To construct a new configuration the resolver searches for modules
in the given parent configurations in the order in which they occur in
the list, and for any given parent it searches its parents recursively in
the same fashion, before moving on to the next parent in the list.  This
search order is, thus, not only depth-first but also determined by the
order of the parents in each list.  The latter property makes it easy to
tell when one candidate module will shadow another.

To instantiate a new layer from a given configuration, the list of the
configurations of its parent layers must be identical to the list of the
parents of that configuration.

The service providers for a module in a given layer are located in the
same way that the resolver searches for modules, i.e., depth-first and
in list order.


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/#NonHierarchicalLayers

From mark.reinhold at oracle.com  Mon Oct 31 21:31:39 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Mon, 31 Oct 2016 14:31:39 -0700
Subject: Issue list and draft JLS/JVMS text updated
Message-ID: <20161031143139.779016730eggemoggin.niobe.net>

FYI, I've updated the issue list to link to the latest proposals for
#ReflectiveAccessToNonExportedTypes and #AwkwardStrongEncapsulation,
to link to the first proposal for #NonHierarchicalLayers, and to
include the new issue #IndirectQualifiedReflectiveAccess.  (I added
#DiscardableModules and #ReadabilityAddedByLayerCreator last week.)

  http://openjdk.java.net/projects/jigsaw/spec/issues/

I've also updated the draft JLS/JVMS text to include changes for the
open-modules/packages proposal:

  http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html

- Mark