From david.lloyd at redhat.com  Tue Sep  6 14:24:54 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Tue, 6 Sep 2016 09:24:54 -0500
Subject: Discussion: #ResourceExistenceAndSize
In-Reply-To: <20160719210808.6AA72B9510@eggemoggin.niobe.net>
References: <56D85B77.4040607@redhat.com>
	<20160713142913.C1C1FB78C7@eggemoggin.niobe.net>
	<57866AD3.4020807@redhat.com>
	<20160719210808.6AA72B9510@eggemoggin.niobe.net>
Message-ID: <1ced87cf-6f70-e49a-332a-5da39f0d68b6@redhat.com>

On 07/19/2016 04:08 PM, mark.reinhold at oracle.com wrote:
> 2016/7/13 9:22:43 -0700, david.lloyd at redhat.com:
>> On 07/13/2016 09:29 AM, mark.reinhold at oracle.com wrote:
>>> Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#ResourceExistenceAndSize
>>>
>>> ...
>>>
>>> At any rate, I don't see anything here that's inherently part of the
>>> design of the module system, so I intend to close this issue as
>>> unrelated.
>>
>> The relation comes in because Class and ClassLoader have a way to open a
>> resource with a URL, but Module does not, providing only a
>> getResourceAsStream() method.  Adding a getResource() method would be
>> one solution, but the underlying requirement is really that the resource
>> existence and size (and origin) could/should be queriable as well, so
>> that was the issue I registered, especially as URL is really not a great
>> solution to this problem.
>
> So would a simple Module::getResourceSize(String name) method, which
> returns a long value to indicate the size of the named resource, or
> else -1 if it doesn't exist, be sufficient?
>
> We could consider adding a similar method alongside the other existing
> getResource* methods in java.lang.Class and ClassLoader.

This would be minimally acceptable.  I think having some kind of handle 
to a resource is "better" for various reasons - particularly where class 
loaders are involved, given the arbitrary nature of class loader 
delegation, so that when one resource references another, the loading 
party can ensure that the intent of that reference is made clear by 
initiating related loads from the same "vector", be it a module or a 
class loader.  URL is clearly an inadequate tool for this purpose for 
reasons which are well-understood I think, hence my proposal for a 
"resource" concept.  This would solve the size/existence problem as well 
as the initial vector problem, and should be feasible to implement.
-- 
- DML

From david.lloyd at redhat.com  Tue Sep  6 14:36:21 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Tue, 6 Sep 2016 09:36:21 -0500
Subject: Resolution to #ClassLoaderNames
Message-ID: <c8557c68-5770-bb3a-ea80-78a13b27058f@redhat.com>

This past July the #ClassLoaderNames issue was raised with a proposed 
solution which we accepted.  Just wanted to follow up to ask: is there a 
reason it hasn't been marked as resolved (e.g. implementation problem, 
waiting for discussion, etc.)?

Thanks.
-- 
- DML

From Tim_Ellison at uk.ibm.com  Wed Sep  7 13:57:55 2016
From: Tim_Ellison at uk.ibm.com (Tim Ellison)
Date: Wed, 7 Sep 2016 14:57:55 +0100
Subject: Artical on JPMS integration with existing module systems
In-Reply-To: <OF349B4469.ABDAB9AA-ON00258013.004B08BD-86258013.004B8BD5@notes.na.collabserv.com>
References: <OF349B4469.ABDAB9AA-ON00258013.004B08BD-86258013.004B8BD5@notes.na.collabserv.com>
Message-ID: <OFB9D82F52.23C5B2F8-ON80258027.0045F544-80258027.004CB66A@notes.na.collabserv.com>

Obviously Tom has come at this integration challenge from his deep
knowledge of a particular OSGi implementation.  There are some
important observations, and minor tweaks to the current model that
will improve JPMS.

"Thomas Watson" <tjwatson at us.ibm.com> wrote on 18/08/2016 14:45:06:
> Below is the plain text format from an article I wrote on using JPMS 
> layers with an existing module system (OSGi).  Please excuse the poor 
> formatting.  I tried to make it is pretty as I could in plain text to 
> allow Mailman to let it through.  I have omitted the diagrams to avoid 
> this message getting held up by Mailman also.  If the diagrams are 
> important I can post them some other way later:

Disclosure: I read the html version, and trust that this plain text is a
faithful reproduction.

I'm going to snip out most of the original post too, which of course is
required reading to get the context and details for this experiment.
However, I want to focus on the specific recommendations for this group,
so I will skip straight to the conclusions.

<snip/> 

> Conclusion

Ok, this is where there are a few issues and recommendations that this
group can debate...

> This approach allows for a pretty accurate representation of a static 
set 
> of resolved OSGi bundles as JPMS modules.  But we are left with several 
> issues that need to be addressed before this can be considered a truly 
> viable solution.  Some may decide these are permanent restrictions of 
JPMS 
> that we will have to live with going forward.  But I believe there are 
> some tweaks to JPMS that could go a long ways to making this approach 
> close to a complete solution.  Listed below are some changes that would 
> help.  I listed them in the order of importance, but I think 1 and 2 are 
a 
> close tie for most important.
> 
>     1. Allow for code that manages a JPMS layer to have more control for 

> establishing read access for the modules contained in the managed layer. 

> The Module addReads method allows for read access to be added for a 
module 
> dynamically at runtime.  But it has a restriction that it must be called 

> by a class defined by the module that wants new read access.  It would 
be 
> a great help if we could call addReads from the management code that 
> created the layer.  Perhaps an addReads(Module wantsRead, Module 
toTarget) 
> method on Layer that checks the caller module is the same module get 
> created the Layer?  This could be used to solve a large set of issues 
> outlined above:

That sounds like a reasonable request.

>         - JPMS-ISSUE-003 - We could avoid having to make JPMS aware of 
the 
> OSGI dependencies if we would be allowed to establish the read access 
> ourselves when the bundle layer is created.
> 
>         - JPMS-ISSUE-004 - If we avoid having to make JPMS aware of the 
> OSGi dependencies then we no longer have worry about restricting cycles.
> 
>         - JPMS-ISSUE-005 - If we can dynamically add reads then we can 
> enable dynamic package import to work by dynamically adding read access 
to 
> the provider of the package at runtime.
> 
>         - JPMS-ISSUE-008 - If we avoid having to make JPMS aware of the 
> OSGi dependencies then we no longer have to worry about restricting 
split 
> packages.
> 
>     2. Allow for reflection on classes from concealed packages.  Many 
> dependency injection containers depend on being able to act upon 
concealed 
> classes in order to construct objects and inject the objects with 
> dependencies.  Forcing implementation details to be exported so that 
these 
> classes can be acted upon by DI containers is wrong.

I believe this is already covered by #ReflectiveAccessToNonExportedTypes
http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes

>         - JPMS-ISSUE-001 - We would no longer have to declare the bundle 

> private packages as exported by the JPMS module.  Instead they can 
remain 
> concealed as they should be.
> 
>     3. Allow for sub-graphs of modules to be discarded within a layer.

Again, this sounds like a reasonable request.

>         - JPMS-ISSUE-007 - This would allow us to flush out the "dead" 
> modules which should never be used anymore.
> 
>     4. Allow a layer to map a class loader to a default named module. 
Any 
> classes from unknown packages to the JPMS would be assigned this named 
> module instead of the unnamed module.
> 
>         - JPMS-ISSUE-002 - This would allow us to avoid having to scan 
for 
> private packages.  Instead we would map the bundle classloader to a 
module 
> and that module could be used for the private packages.
> 
>     5. Allow the JPMS requires statement to specify a module version.

Consideration of versioning was explicitly excluded from the original
requirements, and I don't see it coming in at this stage.  If you are 
controlling
the resolution of modules then you would be able to take account of 
version
numbers in the integration.

>         - JPMS-ISSUE-006 - This would allow us to represent multiple 
> versions of a bundle within the bundle layer and give JPMS modules the 
> ability to specify which version they want.
> 
> My hope is that this experiment is useful in providing constructive 
> feedback to the JPMS expert group.  I hope they consider enhancing JPMS 
to 
> make JPMS layers more usable with existing module systems like OSGi.

Most of the suggestions are modest enhancements to the current model, so 
it
is encouraging to see that a good level of interoperability can be 
obtained
with a few minor tweaks.

Thanks for doing this experiment Tom.  I look forward to others' comments 
too.

> This was extracted from my article at: 
> http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html

Regards,
Tim



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

From mark.reinhold at oracle.com  Sun Sep 11 21:24:57 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Sun, 11 Sep 2016 14:24:57 -0700
Subject: Closing out some open issues
Message-ID: <20160911142457.89623900eggemoggin.niobe.net>

Proposals for the following issues have been available for evaluation
and experimentation for quite a while now.  Most responses have been
positive and there have been no strong objections, so I've updated
the issue list [1] to mark them as closed.

  #BootstrapClassLoaderSearchInJVMTI
  #ClassFileAccPublic
  #CompileTimeDependences (`requires static`)
  #CustomizableAutomaticModuleNameMapping
  #ModuleAnnotations
  #ModuleDeprecation
  #ReflectiveAccessByInstrumentationAgents

Not everyone was thrilled with the choice of `static` as the modifier
on `requires` directives that indicates a compile-time dependence, but
no obviously-better choice has emerged.

- Mark


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/

From forax at univ-mlv.fr  Sun Sep 11 22:04:45 2016
From: forax at univ-mlv.fr (Remi Forax)
Date: Mon, 12 Sep 2016 00:04:45 +0200 (CEST)
Subject: Closing out some open issues
In-Reply-To: <20160911142457.89623900eggemoggin.niobe.net>
References: <20160911142457.89623900eggemoggin.niobe.net>
Message-ID: <301259908.5830.1473631485415.JavaMail.zimbra@u-pem.fr>

i agree.

R?mi

----- Mail original -----
> De: "mark reinhold" <mark.reinhold at oracle.com>
> ?: jpms-spec-experts at openjdk.java.net
> Envoy?: Dimanche 11 Septembre 2016 23:24:57
> Objet: Closing out some open issues

> Proposals for the following issues have been available for evaluation
> and experimentation for quite a while now.  Most responses have been
> positive and there have been no strong objections, so I've updated
> the issue list [1] to mark them as closed.
> 
>  #BootstrapClassLoaderSearchInJVMTI
>  #ClassFileAccPublic
>  #CompileTimeDependences (`requires static`)
>  #CustomizableAutomaticModuleNameMapping
>  #ModuleAnnotations
>  #ModuleDeprecation
>  #ReflectiveAccessByInstrumentationAgents
> 
> Not everyone was thrilled with the choice of `static` as the modifier
> on `requires` directives that indicates a compile-time dependence, but
> no obviously-better choice has emerged.
> 
> - Mark
> 
> 
> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/

From mark.reinhold at oracle.com  Mon Sep 12 15:07:01 2016
From: mark.reinhold at oracle.com (Mark Reinhold)
Date: Mon, 12 Sep 2016 08:07:01 -0700 (PDT)
Subject: More proposals for open issues
Message-ID: <20160912150701.A8934CD07B@eggemoggin.niobe.net>

I'm about to post proposals, or revised proposals, for the following
issues:

  #ReflectiveAccessToNonExportedTypes & #AwkwardStrongEncapsulation
  #ResourceEncapsulation & #ClassFilesAsResources
  #AddExportsInManifest
  #VersionsInModuleNames
  #ClassLoaderNames
  #ServiceLoaderEnhancements

I'll update the issue list [1] immediately thereafter with links to the
proposal messages.

Be warned that the proposal for #ReflectiveAccessToNonExportedTypes and
#AwkwardStrongEncapsulation makes significant changes to the syntax and
semantics of module declarations.  The latter issue is a new issue,
reported to me in person at JVMLS last month by Martin Buchholz and in
e-mail by Aleksey Shipilev, and is summarized thus:

  #AwkwardStrongEncapsulation --- A non-public element of an exported
  package can still be accessed via the `AccessibleObject::setAccessible`
  method of the core reflection API.  The only way to strongly
  encapsulate such an element is to move it to a non-exported package.
  This makes it awkward, at best, to encapsulate the internals of a
  package that defines a public API.

The present design suffers this limitation due to an earlier compromise
made to accommodate reflective access by frameworks, but experience has
now shown that to be a problematic approach.  It would be unfortunate
indeed to bake this limitation into the module system for all time: It
would make it much more difficult for developers to strongly encapsulate
the internals of their own modules, yet enabling such encapsulation is
one of the primary goals of this JSR.  Thus this new proposal, which
introduces the concepts of weak modules and private exports and removes
the previously-proposed notion of dynamic exports.  This has taken some
time to work out but, in the end, appears to achieve a better balance
of usability, ease of migration, and expressive power.

As usual, if you comment on one of these proposals then please either
reply on the existing thread or else include the hashtag of the relevant
issue(s) in the subject line of a new thread, to simplify tracking.

I look forward to your feedback!

- Mark


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/

From mark.reinhold at oracle.com  Mon Sep 12 15:08:01 2016
From: mark.reinhold at oracle.com (Mark Reinhold)
Date: Mon, 12 Sep 2016 08:08:01 -0700 (PDT)
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &
	#AwkwardStrongEncapsulation: Weak modules & private exports
Message-ID: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>

Issue summary
-------------

  #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
  libraries require reflective access to members of the non-exported
  types of other modules; examples include dependency injection (Guice),
  persistence (JPA), debugging tools, code-automation tools, and
  serialization (XStream).  In some cases the particular library to be
  used is not known until run time (e.g., Hibernate and EclipseLink both
  implement JPA).  This capability is also sometimes used to work around
  bugs in unchangeable code.  Access to non-exported packages can, at
  present, only be done via command-line flags, which is extremely
  awkward.  Provide an easier way for reflective code to access such
  non-exported types. [1]

  #AwkwardStrongEncapsulation --- A non-public element of an exported
  package can still be accessed via the `AccessibleObject::setAccessible`
  method of the core reflection API.  The only way to strongly
  encapsulate such an element is to move it to a non-exported package.
  This makes it awkward, at best, to encapsulate the internals of a
  package that defines a public API. [2]

Proposal
--------

(Warning: This is somewhat long, and in the end it affects both `exports`
 and `requires` directives.)

Extend the language of module declarations with the concept of _weak
modules_.  Weak modules make it easy to modularize components whose
internals will be accessed by reflection-based frameworks.  Every
package in a weak module has the following properties:

  (A) It is exported at both compile time and run time, as if by an
      `exports` directive, and

  (B) Its non-public elements are available for _deep_ reflection, i.e.,
      at run time they can be made accessible to code outside the module
      via the `AccessibleObject::setAccessible` method of the core
      reflection API.

In other words, every type defined in a weak module, whether public or
not, is subject to exactly the same access checks as in Java SE 8 and
earlier releases.

A weak module is declared by placing the modifier `weak` before the
`module` keyword.  The declaration of a weak module cannot contain any
explicit `exports` directives.  If the `weak` modifier does not appear
before the `module` keyword then the declared module is _strong_, and
it can contain explicit `exports` directives.

Suppose we have a module `foo.bar` which has an internal package
`com.foo.bar.model` that contains entity classes to be manipulated by
Hibernate, via core reflection.  Then the module declaration

    weak module foo.bar {
        // No exports
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

exports the public types in `com.foo.bar.model`, and those of any other
packages, in all phases.  It additionally makes all non-public elements
of all packages available for deep reflection, enabling Hibernate to
access such elements in the `com.foo.bar.model` package via the
`setAccessible` method.

Weak modules simplify the process of migrating to modules.  The steps
to convert an existing component into a module were, previously:

  (1) Make any changes necessary to get it working as an automatic
      module (e.g., eliminate duplicate packages), and then

  (2) Write an explicit module declaration, which entails identifying
      both the component's dependences (`requires`) and the packages
      whose public types are to be made available to other modules
      (`exports`).

With weak modules we can now divide the second step into two steps:

  (2a) Write an explicit module declaration for a weak module, which
       entails identifying just the component's dependences (`requires`).

  (2b) Convert the weak module into a strong module, which entails
       identifying the packages of the component whose public types
       are to be made available to other modules (`exports`).

In other words, weak modules make it possible to focus first upon the
reliable configuration of a module (`requires`), and then later think
about its strong encapsulation (`exports`).

Weak modules are "weak" in what they export, but they remain subject
to all of the constraints required to achieve reliable configuration.
They do not read the unnamed module (i.e., the class path), they do not
allow cycles in the module graph, and they do not allow split packages.
Weak modules read named modules only as indicated by their `requires`
directives, and they consume and provide services only as indicated by
their `uses` and `provides` directives.

                                  * * *

In a strong module, an ordinary `exports` directive exports a package at
both compile time and run time (property (A) above) but does not make its
non-public types available for deep reflection (B).  In order to enable a
package in a strong module to be exported in the same way as in a weak
module we introduce the per-export modifier `private` to denote this
second property.

If the above weak `foo.bar` module, e.g., contains some other packages
besides `com.foo.bar.model`, and we wish to encapsulate those packages,
we can convert it into a strong module with the declaration

    module foo.bar {
        exports private com.foo.bar.model;
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

Now Hibernate can still access any public or non-public entity classes in
the `com.foo.bar.model` package, but all the other packages are strongly
encapsulated.

The `private` modifier should generally not be used to export a package
containing an API, since normally an API's internal implementation
details should be strongly encapsulated.  It may, however, be useful for
legacy APIs whose internals are known to be accessed by existing code.

Every package in a weak module, an automatic module, or an unnamed module
is exported as if by an `exports private` directive.

To ensure the integrity of the platform we expect few, if any, packages
in the JDK itself to be exported with the `private` modifier.

                                  * * *

The new `private` modifier can also be used with qualified exports,
though they interact with unqualified exports in a non-obvious way.

  - If you write `exports p` then you can also write `exports private
    p to m`, so that code in module `m` can access the non-public types
    of `p` via deep reflection but code outside of `m` can only access
    the public types of `p`.

  - If you write `exports p to m1` then you can also write `exports
    private p to m2`, so that code in `m2` can access the non-public
    types of `p` via deep reflection, code in `m1` can access the
    public types of `p`, but no code in any other module can access
    any of the types of `p`.

  - If you write `exports private p` then you cannot also have a
    qualified export of `p`, since code in all other modules already
    has access to the non-public types of `p` via deep reflection.

Put informally, you can give your friends additional access, but you
can't discriminate against them by giving them less access than everyone
else.

As before, duplicate `exports` directives are not permitted, in order to
ensure easy readability.  At most one `exports` directive is relevant to
any given package/module pair, and it's easy to determine which one.

                                  * * *

The introduction of `private` as a modifier of `exports` directives calls
the existing syntax of `requires public` even more strongly into question
than before.  A module declaration of the form

    module foo.bar {
        exports private com.foo.bar.baz;
        requires public java.sql;
    }

is likely to be very confusing to an uninformed reader.  The `private`
modifier in the `exports` directive means that the private elements of
the `com.foo.bar.baz` package are exported for deep reflection at run
time.  The `public` modifier in the `requires` directive, however, does
not mean that the public elements of the `java.sql` module are needed by
this module; that is true of any plain `requires` directive.  It means
that, additionally, any client of this module is granted implied
readability to the `java.sql` module, thereby gaining access to all of
its exported types.

To reduce this confusion we rename the `public` modifier in `requires`
directives to `transitive`.  Thus the above example becomes

    module foo.bar {
        exports private com.foo.bar.baz;
        requires transitive java.sql;
    }

This is potentially confusing in a different way, since in mathematics
the term "transitive" is usually applied to an entire relation rather
than to three specific elements of a set.  Its use here does not, in
particular, mean that the resolver does not interpret plain `requires`
directives when computing the transitive closure of a set of root
modules.  "Transitive" as used here is in the more abstract sense,
expressing the notion of conveying a property -- in this case, the
readability of the required module -- from one thing to another.


Notes
-----

  - This is significantly different from the first proposal [3].  It adds
    the notion of weak modules, to ease migration, and also the notion of
    exporting a package without enabling deep reflection, to strengthen
    encapsulation.

  - This proposal removes the notion of dynamic exports, which in the
    presence of private exports would introduce considerable complexity
    into the interactions between qualified and unqualified exports.
    This means that it is no longer possible to export a package only at
    run time, so it is no longer possible for the author of a module to
    express the intent that the types of a non-API package are meant to
    be available to frameworks for deep reflection at run time but
    inaccessible at compile time.  The dynamic-export feature could, if
    needed, be added in a future release.

  - A strong module with no exports makes no types accessible to code in
    other modules while a weak module makes all of its types accessible,
    both directly and via deep reflection.  The declarations of such
    modules are, however, visually similar since most of their text lies
    between the curly braces:

        module m1 {
            requires ...;
            uses ...;
            provides ...;
        }

        weak module m2 {
            requires ...;
            uses ...;
            provides ...;
        }

    We suspect that this visual similarity will not cause much confusion
    in practice since strong modules that export no packages will be very
    rare.

  - If a container is to ensure that a package in an application module
    is available for deep reflection only by a trusted framework then it
    can arrange for that by rewriting that module's descriptor, as
    suggested previously [4], to insert the appropriate qualified private
    export.  If there is a possibility that two modules will need to
    export a package of the same name to the same framework module, as
    suggested by Jason Greene [5], then the container should instead
    inject a small class into each module whose static initializer
    invokes the `Module::addExports` method in order to export the
    package to the framework module.  There is no need any longer for
    the resolution algorithm to take this scenario into account [6].

  - This proposal primarily addresses "friendly" uses of reflection, such
    as dependency injection and persistence, in which the author of a
    module knows in advance that one or more packages must be exported at
    run time for deep reflective access by frameworks.  Intrusive access
    to arbitrary packages of arbitrary modules by, e.g., serialization
    frameworks or debugging tools, will still require the use of sharp
    knives such as the `--add-exports` command-line option, the legacy
    unsupported `sun.misc.Unsafe` API and related APIs, or JVM TI.

  - Using the `--add-exports` option or its equivalent remains awkward,
    and sometimes it's the only way out.  To ease migration it's worth
    considering some way for an application packaged as a JAR file to
    include such options in its `MANIFEST.MF` file, as suggested by Simon
    Nash [7].  This is tracked as #AddExportsInManifest [8].


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
[2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
[3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
[4] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
[5] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
[6] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
[7] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
[8] http://openjdk.java.net/projects/jigsaw/spec/issues/#AddExportsInManifest

From mark.reinhold at oracle.com  Mon Sep 12 15:09:01 2016
From: mark.reinhold at oracle.com (Mark Reinhold)
Date: Mon, 12 Sep 2016 08:09:01 -0700 (PDT)
Subject: Proposal: #AddExportsInManifest
Message-ID: <20160912150901.B13EACD084@eggemoggin.niobe.net>

Issue summary
-------------

  #AddExportsInManifest -- Using a command-line option such as
  `--add-exports` to make JDK-internal APIs available to existing code is
  difficult, if not impossible, for applications that are delivered as
  executable JAR files.  To ease migration, consider allowing such a JAR
  file to include the equivalent of such options in its `MANIFEST.MF`
  file. [1]

Proposal
--------

Define two new JAR-file manifest attributes [2], `Add-Exports` and
`Add-Exports-Private`.  Each attribute can occur at most once, in the
main section of a `MANIFEST.MF` file.  The parameter of each attribute is
a space-separated list of slash-separated module-name/package-name pairs.
At run time the presence of a pair `m/p` in the value of each attribute
is interpreted as follows:

  - If `m/p` is listed in the value of the `Add-Exports` attribute then
    package `p` of module `m` is exported to all unnamed modules, as if
    by invoking the `Module::addExports` method.  This makes all of
    package `p`'s public elements immediately accessible to code in
    unnamed modules.

  - If `m/p` is listed in the value of the `Add-Exports-Private`
    attribute then package `p` of module `m` is exported to all unnamed
    modules and its non-public elements are made available for deep
    reflection, as if by invoking the `Module::addExportsPrivate` method
    [3].  This makes all of package `p`'s public elements immediately
    accessible to code in the unnamed module, and all of its non-public
    elements accessible to code in unnamed modules via the
    `setAccessible` method of the core reflection API.

In contrast to language-level module declarations some sloppiness is
allowed in the values of these attributes, as is traditional in JAR-file
manifests.  A particular module-name/package-name pair can be listed more
than once in either of these attributes.  It can also be listed in both
attributes, in which case the package will be exported privately.  A
module or package specified in such a pair need not actually exist in
the run-time module graph.

These attributes are interpreted only in the main executable JAR file
of an application, i.e., in the JAR file specified to the `-jar`
command-line option of typical Java run-time launchers.  They are
ignored in all other JAR files.

As an example, the Spring Boot framework [4] uses `setAccessible` to
access a protected `defineClass` method of `java.lang.ClassLoader` and
a private constructor of `java.lang.invoke.MethodHandles.Lookup`.  An
application that uses Spring Boot can be made to work by adding the
line

    Add-Exports-Private: java.base/java.lang java.base/java.lang.invoke

to the `MANIFEST.MF` file of its executable JAR file.


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/#AddExportsInManifest
[2] http://download.java.net/java/jdk9/docs/technotes/guides/jar/jar.html#JAR_Manifest
[3] See the nearby proposal for #ReflectiveAccessToNonExportedTypes
[4] http://projects.spring.io/spring-boot/

From mark.reinhold at oracle.com  Mon Sep 12 15:10:01 2016
From: mark.reinhold at oracle.com (Mark Reinhold)
Date: Mon, 12 Sep 2016 08:10:01 -0700 (PDT)
Subject: Proposal: #ResourceEncapsulation & #ClassFilesAsResources (revised)
Message-ID: <20160912151001.B5956CD089@eggemoggin.niobe.net>

Issue summaries
---------------

  #ClassFilesAsResources --- If a type is visible and was loaded from a
  class file then it should be possible to read that file by invoking the
  `getResourceAsStream` method of the type's class loader, as it is in
  earlier releases. [1]

  #ResourceEncapsulation --- The `Module::getResourceAsStream` method can
  be used to read the resources of any named module, without restriction,
  which violates the resource-encapsulation requirement [2].  This method
  should be restricted somehow so that only "suitably-privileged" code
  (for some definition of that term) can access resources in a named
  module other than its own.  An alternative is to drop this
  requirement. [3]

Proposal
--------

Extend the notion of private exports, introduced nearby in the revised
proposal for #ReflectiveAccessToNonExportedTypes, to govern whether a
resource defined in a named module can be located by code in some other
module.

Resources are named by URL-like path strings.  The _effective package
name_ of an absolute resource name is computed by removing the initial
slash character, removing the final slash character (`'/'`) and any
subsequent characters, and then converting any remaining slash characters
to periods (`'.'`).  The effective package name of a resource named by
the string `"/foo/bar/baz"`, e.g., is 'foo.bar'.  The effective package
name of a relative resource name is computed by resolving the resource
name against a particular class, as if by the `Class::getResource`
method, and then computing the effective package name of the result.

For a resource defined in a named module we impose the following
restrictions upon code in other modules:

  - If a resource's name ends in `".class"` then it can be located by
    code in any module.

  - If a resource's effective package name is not a valid Java language
    package name (e.g., `"META-INF.foo.bar"`) [4] then the resource can
    be located by code in any module.

  - If the package is exported privately, without qualification, then the
    resource can be located by code in any module.

  - If the package is exported privately to a specific set of modules
    then the resource can be located by code in those modules.

  - If the package is not exported privately in any way then the resource
    cannot be located by code outside of the module itself.

We revise the various `getResource*` methods to impose the above
constraints:

  - The `Module::getResourceAsStream` method allows a resource to be
    located directly within a module, without first having to load a
    class from that module.  This method is caller-sensitive.

  - The `Class::getResource*` methods, when invoked upon a class defined
    in a named module, only locate resources from within that module.
    These methods are also caller-sensitive.

  - The `ClassLoader::getResource*` methods will locate resources in
    named modules subject to the above restrictions except that the
    effective package of the resource must be exported privately without
    qualification, since this method is not caller-sensitive.  Custom
    class loaders that locate resources in modules should implement these
    restrictions, but there is no way to force them to do so.  The order
    of the elements of an enumeration returned by the `getResources`
    method is unspecified, but for a given class loader `cl` the values
    of `cl.getResources(name).nextElement()` and `cl.getResource(name)`
    are always equal.

Every package in an unnamed, automatic, or weak module is exported
privately, so all of the resources in such modules can be located by
code in all other modules.

Notes
-----

  - The first proposal for these issues [5] suggested that we simply drop
    the agreed resource-encapsulation requirement [2].  As suggested both
    in the EG [6] and elsewhere [7][8], this revised proposal attempts to
    strike a balance between practical compatibility and modular
    encapsulation.

  - This proposal leverages private exports, i.e., the `exports private`
    directive, since the internal resources of a module are much like the
    non-public elements of the types defined in a module.  This avoids
    leaking resources inadvertently from packages that are exported
    normally, i.e., without the `private` modifier.  It also avoids the
    need to extend the syntax and semantics of module declarations with
    a completely new directive just for resources, which seems like
    overkill.


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassFilesAsResources
[2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#resource-encapsulation
[3] http://openjdk.java.net/projects/jigsaw/spec/issues/#ResourceEncapsulation
[4] http://docs.oracle.com/javase/specs/jls/se8/html/jls-6.html#jls-PackageName
[5] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000309.html
[6] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000322.html
[7] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-June/000050.html
[8] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-June/000051.html

From mark.reinhold at oracle.com  Mon Sep 12 15:11:01 2016
From: mark.reinhold at oracle.com (Mark Reinhold)
Date: Mon, 12 Sep 2016 08:11:01 -0700 (PDT)
Subject: Proposal: #VersionsInModuleNames
Message-ID: <20160912151101.BA3D0CD08E@eggemoggin.niobe.net>

Issue summary
-------------

  #VersionsInModuleNames --- Some have argued that library maintainers
  will be tempted to encode major version numbers, or even full version
  numbers, in module names.  Is there some way we can guide people away
  from doing that? [1]

Proposal
--------

Make two small changes:

  - Revise the automatic-module naming algorithm implemented by `javac`
    at compile time and the `ModuleFinder::of` method [2] at run time.
    It will now strip any trailing digits and period characters that
    remain after removing the version component, if any, from the name
    of the original JAR file.  Thus `foo-bar-1.2.3.jar` becomes the
    automatic module named `foo.bar` with the version string `1.2.3`,
    and `foo-bar42.jar` becomes the automatic module named `foo.bar`
    with no version string.

  - Revise `javac` to emit a lint warning, enabled by default, when a
    module-declaration compilation unit mentions a module name that
    ends with one or more digit characters, whether it is the name of
    the module being declared or the name of a module referenced in a
    `requires` directive or a qualified `exports` directive.


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/#VersionsInModuleNames
[2] http://download.java.net/java/jigsaw/docs/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path...-

From mark.reinhold at oracle.com  Mon Sep 12 15:12:01 2016
From: mark.reinhold at oracle.com (Mark Reinhold)
Date: Mon, 12 Sep 2016 08:12:01 -0700 (PDT)
Subject: Proposal: #ClassLoaderNames
Message-ID: <20160912151201.BED27CD092@eggemoggin.niobe.net>

Issue summary
-------------

  #ClassLoaderNames --- Enhance class loaders to have optional names, so
  that external module systems can provide better diagnostics.  When the
  run-time system generates a stack trace or an exception message that
  mentions a module name and version, if present, then it should also
  insert the name of that module's class loader, if present. [1]

Proposal
--------

Add a string-returning `getName()` method to `java.lang.ClassLoader`
along with a new constructor with which the name can be specified.  If a
name is not specified when a class loader is created then it will have no
name, i.e., the `getName` method will return `null`.  Add corresponding
constructors to `java.net.URLClassLoader` and
`java.security.SecureClassLoader`.

Extend `java.lang.StackTraceElement` so that a stack-trace element can
include the name of the relevant class loader, and revise its `toString`
method to convey that name when present.

Arrange for the built-in platform and application class loaders to have
the names `"platform"` and `"app"`, respectively.  (The bootstrap class
loader cannot be given a name since it is not reified as a `ClassLoader`
object.)


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassLoaderNames

From mark.reinhold at oracle.com  Mon Sep 12 15:13:01 2016
From: mark.reinhold at oracle.com (Mark Reinhold)
Date: Mon, 12 Sep 2016 08:13:01 -0700 (PDT)
Subject: Proposal: #ServiceLoaderEnhancements
Message-ID: <20160912151301.C2B83CD096@eggemoggin.niobe.net>

Issue summary
-------------

  #ServiceLoaderEnhancements --- The module system encourages the use of
  services for loose coupling, but the `ServiceLoader` class is not very
  flexible.  Consider enhancing it so that (1) neither a provider class
  nor its no-args constructor need be declared `public`, (2) a provider
  can be a singleton, or perhaps a collection of singletons, and (3) the
  classes of the available providers can be inspected and selected prior
  to instantiation. [13]

Proposal
--------

  (1) No change: Continue to require service-provider classes, and their
      no-args constructors, to be public.

      Providers on the class path, and their no-args constructors, must
      always be public.  Allowing a class-path provider or its no-args
      constructor to be non-public introduces a security risk, since an
      adversary could place a `META-INF/services` entry elsewhere on the
      class path in order to force that otherwise-inaccessible
      constructor to be invoked.

      For providers in named modules, allowing non-public provider
      classes and non-public no-args constructors isn't really necessary
      and is, in some ways, counterproductive.  In a named module a
      provider class, and its constructor, can be encapsulated by placing
      the provider in an unexported package.  Having to declare the
      provider class and its no-args constructor `public` is a useful
      declaration of intent, since they will be accessed by the service
      loader itself, and hence useful documentation.

  (2) Revise the `ServiceLoader` class so that if a candidate provider
      class in a named module has a no-args public static method named
      `provider` then that method is invoked and its result is taken as
      the provider object.  An exception is thrown if the method does
      not have an appropriate return type.  A static `provider` method
      can either return a singleton or act as a factory method.  If the
      candidate provider class does not have such a method then its
      public no-args constructor, if any, is invoked, per (1) above.

      (An alternative is to use an annotation, say `@Provider`, to
       identify provider-containing fields or provider-returning methods.
       The cost of loading the annotation-reading code into the JVM is,
       however, nontrivial, and since services are used widely within the
       JDK itself we'd prefer not to impose that overhead on all
       applications.)

  (3) Decouple the loading of provider classes from the instantiation of
      such classes: Introduce a new `ServiceLoader.Provider` interface
      that pairs a provider class with a method to instantiate that
      provider, and add a `stream()` method that returns a stream of
      objects implementing that interface.  A client can then filter
      providers by inspecting the elements of the stream, examining each
      provider class and perhaps the annotations thereon, and then
      instantiating the class if appropriate.  (Draft Javadoc source
      attached below.)

[1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ServiceLoaderEnhancements

--

    /**
     * Represents a service provider located by {@code ServiceLoader}.
     *
     * <p> When using a loader's {@link ServiceLoader#stream() stream()} method
     * then the elements are of type {@code Provider}. This allows processing
     * to select or filter on the provider class without instantiating the
     * provider. </p>
     *
     * @param  <S> The service type
     * @since 9
     */
    public static interface Provider<S> extends Supplier<S> {

        /**
         * Returns the provider class. There is no guarantee that this type is
         * accessible and so attempting to instantiate it, by means of its
         * {@link Class#newInstance() newInstance()} method for example, will
         * fail when it is not accessible. The {@link #get() get()} method
         * should instead be used to obtain the provider.
         *
         * @return The provider class
         */
        Class<S> type();

        /**
         * Returns an instance of the provider.
         *
         * @return An instance of the provider.
         *
         * @throws ServiceConfigurationError
         *         If the service provider cannot be instantiated. The error
         *         cause will carry an appropriate cause.
         */
        @Override S get();

    }

    /**
     * Returns a stream that lazily loads the available providers of this
     * loader's service. The stream elements are of type {@link Provider
     * Provider}, the {@code Provider}'s {@link Provider#get() get} method
     * must be invoked to get or instantiate the provider.
     *
     * <p> When processing the stream then providers that were previously
     * loaded by stream operations are processed first, in load order. It then
     * lazily loads any remaining providers. If a provider class cannot be
     * loaded, can't be assigned to the service type, or some other error is
     * thrown when locating the provider then it is wrapped with a {@code
     * ServiceConfigurationError} and thrown by whatever method caused the
     * provider to be loaded. </p>
     *
     * <p> If this loader's provider caches are cleared by invoking the {@link
     * #reload() reload} method then existing streams for this service
     * loader should be discarded. </p>
     *
     * <p> The following examples demonstrate usage. The first example
     * creates a stream of providers, the second example is the same except
     * that it sorts the providers by provider class name (and so locate all
     * providers).
     * <pre>{@code
     *    Stream<CodecSet> providers = ServiceLoader.load(CodecSet.class)
     *            .stream()
     *            .map(Provider::get);
     *
     *    Stream<CodecSet> providers = ServiceLoader.load(CodecSet.class)
     *            .stream()
     *            .sorted(Comparator.comparing(p -> p.type().getName()))
     *            .map(Provider::get);
     * }</pre>
     *
     * @return  A stream that lazily loads providers for this loader's service
     *
     * @since 9
     */
    public Stream<Provider<S>> stream() { ... }

From david.lloyd at redhat.com  Mon Sep 12 15:54:51 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Mon, 12 Sep 2016 10:54:51 -0500
Subject: Proposal: #ClassLoaderNames
In-Reply-To: <20160912151201.BED27CD092@eggemoggin.niobe.net>
References: <20160912151201.BED27CD092@eggemoggin.niobe.net>
Message-ID: <4175aae8-edaa-58ea-271f-ffd40efc2b07@redhat.com>

On 09/12/2016 10:12 AM, Mark Reinhold wrote:
> Issue summary
> -------------
>
>   #ClassLoaderNames --- Enhance class loaders to have optional names, so
>   that external module systems can provide better diagnostics.  When the
>   run-time system generates a stack trace or an exception message that
>   mentions a module name and version, if present, then it should also
>   insert the name of that module's class loader, if present. [1]
>
> Proposal
> --------
>
> Add a string-returning `getName()` method to `java.lang.ClassLoader`
> along with a new constructor with which the name can be specified.  If a
> name is not specified when a class loader is created then it will have no
> name, i.e., the `getName` method will return `null`.  Add corresponding
> constructors to `java.net.URLClassLoader` and
> `java.security.SecureClassLoader`.
>
> Extend `java.lang.StackTraceElement` so that a stack-trace element can
> include the name of the relevant class loader, and revise its `toString`
> method to convey that name when present.
>
> Arrange for the built-in platform and application class loaders to have
> the names `"platform"` and `"app"`, respectively.  (The bootstrap class
> loader cannot be given a name since it is not reified as a `ClassLoader`
> object.)
>
>
> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassLoaderNames

Looks great.  This will be very useful!

-- 
- DML

From david.lloyd at redhat.com  Mon Sep 12 16:01:41 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Mon, 12 Sep 2016 11:01:41 -0500
Subject: Proposal: #VersionsInModuleNames
In-Reply-To: <20160912151101.BA3D0CD08E@eggemoggin.niobe.net>
References: <20160912151101.BA3D0CD08E@eggemoggin.niobe.net>
Message-ID: <ea9f2323-834b-e83e-e46c-76795546ffa1@redhat.com>

On 09/12/2016 10:11 AM, Mark Reinhold wrote:
> Issue summary
> -------------
>
>   #VersionsInModuleNames --- Some have argued that library maintainers
>   will be tempted to encode major version numbers, or even full version
>   numbers, in module names.  Is there some way we can guide people away
>   from doing that? [1]
>
> Proposal
> --------
>
> Make two small changes:
>
>   - Revise the automatic-module naming algorithm implemented by `javac`
>     at compile time and the `ModuleFinder::of` method [2] at run time.
>     It will now strip any trailing digits and period characters that
>     remain after removing the version component, if any, from the name
>     of the original JAR file.  Thus `foo-bar-1.2.3.jar` becomes the
>     automatic module named `foo.bar` with the version string `1.2.3`,
>     and `foo-bar42.jar` becomes the automatic module named `foo.bar`
>     with no version string.
>
>   - Revise `javac` to emit a lint warning, enabled by default, when a
>     module-declaration compilation unit mentions a module name that
>     ends with one or more digit characters, whether it is the name of
>     the module being declared or the name of a module referenced in a
>     `requires` directive or a qualified `exports` directive.
>
>
> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#VersionsInModuleNames
> [2] http://download.java.net/java/jigsaw/docs/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path...-

Seems OK as long as it remains restricted to automatic module naming.
-- 
- DML

From david.lloyd at redhat.com  Mon Sep 12 16:20:31 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Mon, 12 Sep 2016 11:20:31 -0500
Subject: Proposal: #AddExportsInManifest
In-Reply-To: <20160912150901.B13EACD084@eggemoggin.niobe.net>
References: <20160912150901.B13EACD084@eggemoggin.niobe.net>
Message-ID: <b0ff4df2-0638-b3b7-e1cd-e9a9085f4ce6@redhat.com>

On 09/12/2016 10:09 AM, Mark Reinhold wrote:
> Issue summary
> -------------
>
>   #AddExportsInManifest -- Using a command-line option such as
>   `--add-exports` to make JDK-internal APIs available to existing code is
>   difficult, if not impossible, for applications that are delivered as
>   executable JAR files.  To ease migration, consider allowing such a JAR
>   file to include the equivalent of such options in its `MANIFEST.MF`
>   file. [1]
>
> Proposal
> --------
>
> Define two new JAR-file manifest attributes [2], `Add-Exports` and
> `Add-Exports-Private`.  Each attribute can occur at most once, in the
> main section of a `MANIFEST.MF` file.  The parameter of each attribute is
> a space-separated list of slash-separated module-name/package-name pairs.
> At run time the presence of a pair `m/p` in the value of each attribute
> is interpreted as follows:
>
>   - If `m/p` is listed in the value of the `Add-Exports` attribute then
>     package `p` of module `m` is exported to all unnamed modules, as if
>     by invoking the `Module::addExports` method.  This makes all of
>     package `p`'s public elements immediately accessible to code in
>     unnamed modules.
>
>   - If `m/p` is listed in the value of the `Add-Exports-Private`
>     attribute then package `p` of module `m` is exported to all unnamed
>     modules and its non-public elements are made available for deep
>     reflection, as if by invoking the `Module::addExportsPrivate` method
>     [3].  This makes all of package `p`'s public elements immediately
>     accessible to code in the unnamed module, and all of its non-public
>     elements accessible to code in unnamed modules via the
>     `setAccessible` method of the core reflection API.
>
> In contrast to language-level module declarations some sloppiness is
> allowed in the values of these attributes, as is traditional in JAR-file
> manifests.  A particular module-name/package-name pair can be listed more
> than once in either of these attributes.  It can also be listed in both
> attributes, in which case the package will be exported privately.  A
> module or package specified in such a pair need not actually exist in
> the run-time module graph.
>
> These attributes are interpreted only in the main executable JAR file
> of an application, i.e., in the JAR file specified to the `-jar`
> command-line option of typical Java run-time launchers.  They are
> ignored in all other JAR files.
>
> As an example, the Spring Boot framework [4] uses `setAccessible` to
> access a protected `defineClass` method of `java.lang.ClassLoader` and
> a private constructor of `java.lang.invoke.MethodHandles.Lookup`.  An
> application that uses Spring Boot can be made to work by adding the
> line
>
>     Add-Exports-Private: java.base/java.lang java.base/java.lang.invoke
>
> to the `MANIFEST.MF` file of its executable JAR file.

This seems OK I think.

-- 
- DML

From david.lloyd at redhat.com  Mon Sep 12 16:32:17 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Mon, 12 Sep 2016 11:32:17 -0500
Subject: Proposal: #ServiceLoaderEnhancements
In-Reply-To: <20160912151301.C2B83CD096@eggemoggin.niobe.net>
References: <20160912151301.C2B83CD096@eggemoggin.niobe.net>
Message-ID: <f4337544-dd3a-f76c-185c-059fc2603b7f@redhat.com>

On 09/12/2016 10:13 AM, Mark Reinhold wrote:
> Issue summary
> -------------
>
>   #ServiceLoaderEnhancements --- The module system encourages the use of
>   services for loose coupling, but the `ServiceLoader` class is not very
>   flexible.  Consider enhancing it so that (1) neither a provider class
>   nor its no-args constructor need be declared `public`, (2) a provider
>   can be a singleton, or perhaps a collection of singletons, and (3) the
>   classes of the available providers can be inspected and selected prior
>   to instantiation. [13]
>
> Proposal
> --------
>
>   (1) No change: Continue to require service-provider classes, and their
>       no-args constructors, to be public.
>
>       Providers on the class path, and their no-args constructors, must
>       always be public.  Allowing a class-path provider or its no-args
>       constructor to be non-public introduces a security risk, since an
>       adversary could place a `META-INF/services` entry elsewhere on the
>       class path in order to force that otherwise-inaccessible
>       constructor to be invoked.
>
>       For providers in named modules, allowing non-public provider
>       classes and non-public no-args constructors isn't really necessary
>       and is, in some ways, counterproductive.  In a named module a
>       provider class, and its constructor, can be encapsulated by placing
>       the provider in an unexported package.  Having to declare the
>       provider class and its no-args constructor `public` is a useful
>       declaration of intent, since they will be accessed by the service
>       loader itself, and hence useful documentation.
>
>   (2) Revise the `ServiceLoader` class so that if a candidate provider
>       class in a named module has a no-args public static method named
>       `provider` then that method is invoked and its result is taken as
>       the provider object.  An exception is thrown if the method does
>       not have an appropriate return type.  A static `provider` method
>       can either return a singleton or act as a factory method.  If the
>       candidate provider class does not have such a method then its
>       public no-args constructor, if any, is invoked, per (1) above.
>
>       (An alternative is to use an annotation, say `@Provider`, to
>        identify provider-containing fields or provider-returning methods.
>        The cost of loading the annotation-reading code into the JVM is,
>        however, nontrivial, and since services are used widely within the
>        JDK itself we'd prefer not to impose that overhead on all
>        applications.)
>
>   (3) Decouple the loading of provider classes from the instantiation of
>       such classes: Introduce a new `ServiceLoader.Provider` interface
>       that pairs a provider class with a method to instantiate that
>       provider, and add a `stream()` method that returns a stream of
>       objects implementing that interface.  A client can then filter
>       providers by inspecting the elements of the stream, examining each
>       provider class and perhaps the annotations thereon, and then
>       instantiating the class if appropriate.  (Draft Javadoc source
>       attached below.)

I think proposal 2 makes the most sense from our perspective: it's a 
simple, low-infrastructure, effective solution.  I also tend to agree on 
the security points mentioned.

-- 
- DML

From david.lloyd at redhat.com  Mon Sep 12 21:06:43 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Mon, 12 Sep 2016 16:06:43 -0500
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &
	#AwkwardStrongEncapsulation: Weak modules & private exports
In-Reply-To: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
Message-ID: <bbba9e93-fe31-b716-6c0d-95cbbb339cc9@redhat.com>

We are reviewing this internally and will hopefully have feedback soon.

On 09/12/2016 10:08 AM, Mark Reinhold wrote:
> Issue summary
> -------------
>
>   #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
>   libraries require reflective access to members of the non-exported
>   types of other modules; examples include dependency injection (Guice),
>   persistence (JPA), debugging tools, code-automation tools, and
>   serialization (XStream).  In some cases the particular library to be
>   used is not known until run time (e.g., Hibernate and EclipseLink both
>   implement JPA).  This capability is also sometimes used to work around
>   bugs in unchangeable code.  Access to non-exported packages can, at
>   present, only be done via command-line flags, which is extremely
>   awkward.  Provide an easier way for reflective code to access such
>   non-exported types. [1]
>
>   #AwkwardStrongEncapsulation --- A non-public element of an exported
>   package can still be accessed via the `AccessibleObject::setAccessible`
>   method of the core reflection API.  The only way to strongly
>   encapsulate such an element is to move it to a non-exported package.
>   This makes it awkward, at best, to encapsulate the internals of a
>   package that defines a public API. [2]
>
> Proposal
> --------
>
> (Warning: This is somewhat long, and in the end it affects both `exports`
>  and `requires` directives.)
>
> Extend the language of module declarations with the concept of _weak
> modules_.  Weak modules make it easy to modularize components whose
> internals will be accessed by reflection-based frameworks.  Every
> package in a weak module has the following properties:
>
>   (A) It is exported at both compile time and run time, as if by an
>       `exports` directive, and
>
>   (B) Its non-public elements are available for _deep_ reflection, i.e.,
>       at run time they can be made accessible to code outside the module
>       via the `AccessibleObject::setAccessible` method of the core
>       reflection API.
>
> In other words, every type defined in a weak module, whether public or
> not, is subject to exactly the same access checks as in Java SE 8 and
> earlier releases.
>
> A weak module is declared by placing the modifier `weak` before the
> `module` keyword.  The declaration of a weak module cannot contain any
> explicit `exports` directives.  If the `weak` modifier does not appear
> before the `module` keyword then the declared module is _strong_, and
> it can contain explicit `exports` directives.
>
> Suppose we have a module `foo.bar` which has an internal package
> `com.foo.bar.model` that contains entity classes to be manipulated by
> Hibernate, via core reflection.  Then the module declaration
>
>     weak module foo.bar {
>         // No exports
>         requires hibernate.core;
>         requires hibernate.entitymanager;
>     }
>
> exports the public types in `com.foo.bar.model`, and those of any other
> packages, in all phases.  It additionally makes all non-public elements
> of all packages available for deep reflection, enabling Hibernate to
> access such elements in the `com.foo.bar.model` package via the
> `setAccessible` method.
>
> Weak modules simplify the process of migrating to modules.  The steps
> to convert an existing component into a module were, previously:
>
>   (1) Make any changes necessary to get it working as an automatic
>       module (e.g., eliminate duplicate packages), and then
>
>   (2) Write an explicit module declaration, which entails identifying
>       both the component's dependences (`requires`) and the packages
>       whose public types are to be made available to other modules
>       (`exports`).
>
> With weak modules we can now divide the second step into two steps:
>
>   (2a) Write an explicit module declaration for a weak module, which
>        entails identifying just the component's dependences (`requires`).
>
>   (2b) Convert the weak module into a strong module, which entails
>        identifying the packages of the component whose public types
>        are to be made available to other modules (`exports`).
>
> In other words, weak modules make it possible to focus first upon the
> reliable configuration of a module (`requires`), and then later think
> about its strong encapsulation (`exports`).
>
> Weak modules are "weak" in what they export, but they remain subject
> to all of the constraints required to achieve reliable configuration.
> They do not read the unnamed module (i.e., the class path), they do not
> allow cycles in the module graph, and they do not allow split packages.
> Weak modules read named modules only as indicated by their `requires`
> directives, and they consume and provide services only as indicated by
> their `uses` and `provides` directives.
>
>                                   * * *
>
> In a strong module, an ordinary `exports` directive exports a package at
> both compile time and run time (property (A) above) but does not make its
> non-public types available for deep reflection (B).  In order to enable a
> package in a strong module to be exported in the same way as in a weak
> module we introduce the per-export modifier `private` to denote this
> second property.
>
> If the above weak `foo.bar` module, e.g., contains some other packages
> besides `com.foo.bar.model`, and we wish to encapsulate those packages,
> we can convert it into a strong module with the declaration
>
>     module foo.bar {
>         exports private com.foo.bar.model;
>         requires hibernate.core;
>         requires hibernate.entitymanager;
>     }
>
> Now Hibernate can still access any public or non-public entity classes in
> the `com.foo.bar.model` package, but all the other packages are strongly
> encapsulated.
>
> The `private` modifier should generally not be used to export a package
> containing an API, since normally an API's internal implementation
> details should be strongly encapsulated.  It may, however, be useful for
> legacy APIs whose internals are known to be accessed by existing code.
>
> Every package in a weak module, an automatic module, or an unnamed module
> is exported as if by an `exports private` directive.
>
> To ensure the integrity of the platform we expect few, if any, packages
> in the JDK itself to be exported with the `private` modifier.
>
>                                   * * *
>
> The new `private` modifier can also be used with qualified exports,
> though they interact with unqualified exports in a non-obvious way.
>
>   - If you write `exports p` then you can also write `exports private
>     p to m`, so that code in module `m` can access the non-public types
>     of `p` via deep reflection but code outside of `m` can only access
>     the public types of `p`.
>
>   - If you write `exports p to m1` then you can also write `exports
>     private p to m2`, so that code in `m2` can access the non-public
>     types of `p` via deep reflection, code in `m1` can access the
>     public types of `p`, but no code in any other module can access
>     any of the types of `p`.
>
>   - If you write `exports private p` then you cannot also have a
>     qualified export of `p`, since code in all other modules already
>     has access to the non-public types of `p` via deep reflection.
>
> Put informally, you can give your friends additional access, but you
> can't discriminate against them by giving them less access than everyone
> else.
>
> As before, duplicate `exports` directives are not permitted, in order to
> ensure easy readability.  At most one `exports` directive is relevant to
> any given package/module pair, and it's easy to determine which one.
>
>                                   * * *
>
> The introduction of `private` as a modifier of `exports` directives calls
> the existing syntax of `requires public` even more strongly into question
> than before.  A module declaration of the form
>
>     module foo.bar {
>         exports private com.foo.bar.baz;
>         requires public java.sql;
>     }
>
> is likely to be very confusing to an uninformed reader.  The `private`
> modifier in the `exports` directive means that the private elements of
> the `com.foo.bar.baz` package are exported for deep reflection at run
> time.  The `public` modifier in the `requires` directive, however, does
> not mean that the public elements of the `java.sql` module are needed by
> this module; that is true of any plain `requires` directive.  It means
> that, additionally, any client of this module is granted implied
> readability to the `java.sql` module, thereby gaining access to all of
> its exported types.
>
> To reduce this confusion we rename the `public` modifier in `requires`
> directives to `transitive`.  Thus the above example becomes
>
>     module foo.bar {
>         exports private com.foo.bar.baz;
>         requires transitive java.sql;
>     }
>
> This is potentially confusing in a different way, since in mathematics
> the term "transitive" is usually applied to an entire relation rather
> than to three specific elements of a set.  Its use here does not, in
> particular, mean that the resolver does not interpret plain `requires`
> directives when computing the transitive closure of a set of root
> modules.  "Transitive" as used here is in the more abstract sense,
> expressing the notion of conveying a property -- in this case, the
> readability of the required module -- from one thing to another.
>
>
> Notes
> -----
>
>   - This is significantly different from the first proposal [3].  It adds
>     the notion of weak modules, to ease migration, and also the notion of
>     exporting a package without enabling deep reflection, to strengthen
>     encapsulation.
>
>   - This proposal removes the notion of dynamic exports, which in the
>     presence of private exports would introduce considerable complexity
>     into the interactions between qualified and unqualified exports.
>     This means that it is no longer possible to export a package only at
>     run time, so it is no longer possible for the author of a module to
>     express the intent that the types of a non-API package are meant to
>     be available to frameworks for deep reflection at run time but
>     inaccessible at compile time.  The dynamic-export feature could, if
>     needed, be added in a future release.
>
>   - A strong module with no exports makes no types accessible to code in
>     other modules while a weak module makes all of its types accessible,
>     both directly and via deep reflection.  The declarations of such
>     modules are, however, visually similar since most of their text lies
>     between the curly braces:
>
>         module m1 {
>             requires ...;
>             uses ...;
>             provides ...;
>         }
>
>         weak module m2 {
>             requires ...;
>             uses ...;
>             provides ...;
>         }
>
>     We suspect that this visual similarity will not cause much confusion
>     in practice since strong modules that export no packages will be very
>     rare.
>
>   - If a container is to ensure that a package in an application module
>     is available for deep reflection only by a trusted framework then it
>     can arrange for that by rewriting that module's descriptor, as
>     suggested previously [4], to insert the appropriate qualified private
>     export.  If there is a possibility that two modules will need to
>     export a package of the same name to the same framework module, as
>     suggested by Jason Greene [5], then the container should instead
>     inject a small class into each module whose static initializer
>     invokes the `Module::addExports` method in order to export the
>     package to the framework module.  There is no need any longer for
>     the resolution algorithm to take this scenario into account [6].
>
>   - This proposal primarily addresses "friendly" uses of reflection, such
>     as dependency injection and persistence, in which the author of a
>     module knows in advance that one or more packages must be exported at
>     run time for deep reflective access by frameworks.  Intrusive access
>     to arbitrary packages of arbitrary modules by, e.g., serialization
>     frameworks or debugging tools, will still require the use of sharp
>     knives such as the `--add-exports` command-line option, the legacy
>     unsupported `sun.misc.Unsafe` API and related APIs, or JVM TI.
>
>   - Using the `--add-exports` option or its equivalent remains awkward,
>     and sometimes it's the only way out.  To ease migration it's worth
>     considering some way for an application packaged as a JAR file to
>     include such options in its `MANIFEST.MF` file, as suggested by Simon
>     Nash [7].  This is tracked as #AddExportsInManifest [8].
>
>
> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
> [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
> [3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
> [4] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
> [5] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
> [6] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
> [7] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
> [8] http://openjdk.java.net/projects/jigsaw/spec/issues/#AddExportsInManifest
>

-- 
- DML

From david.lloyd at redhat.com  Mon Sep 12 21:06:58 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Mon, 12 Sep 2016 16:06:58 -0500
Subject: Proposal: #ResourceEncapsulation & #ClassFilesAsResources
	(revised)
In-Reply-To: <20160912151001.B5956CD089@eggemoggin.niobe.net>
References: <20160912151001.B5956CD089@eggemoggin.niobe.net>
Message-ID: <12f024c7-07a0-4853-1140-dc367146b035@redhat.com>

We are also reviewing this one internally and will respond soon.

On 09/12/2016 10:10 AM, Mark Reinhold wrote:
> Issue summaries
> ---------------
>
>   #ClassFilesAsResources --- If a type is visible and was loaded from a
>   class file then it should be possible to read that file by invoking the
>   `getResourceAsStream` method of the type's class loader, as it is in
>   earlier releases. [1]
>
>   #ResourceEncapsulation --- The `Module::getResourceAsStream` method can
>   be used to read the resources of any named module, without restriction,
>   which violates the resource-encapsulation requirement [2].  This method
>   should be restricted somehow so that only "suitably-privileged" code
>   (for some definition of that term) can access resources in a named
>   module other than its own.  An alternative is to drop this
>   requirement. [3]
>
> Proposal
> --------
>
> Extend the notion of private exports, introduced nearby in the revised
> proposal for #ReflectiveAccessToNonExportedTypes, to govern whether a
> resource defined in a named module can be located by code in some other
> module.
>
> Resources are named by URL-like path strings.  The _effective package
> name_ of an absolute resource name is computed by removing the initial
> slash character, removing the final slash character (`'/'`) and any
> subsequent characters, and then converting any remaining slash characters
> to periods (`'.'`).  The effective package name of a resource named by
> the string `"/foo/bar/baz"`, e.g., is 'foo.bar'.  The effective package
> name of a relative resource name is computed by resolving the resource
> name against a particular class, as if by the `Class::getResource`
> method, and then computing the effective package name of the result.
>
> For a resource defined in a named module we impose the following
> restrictions upon code in other modules:
>
>   - If a resource's name ends in `".class"` then it can be located by
>     code in any module.
>
>   - If a resource's effective package name is not a valid Java language
>     package name (e.g., `"META-INF.foo.bar"`) [4] then the resource can
>     be located by code in any module.
>
>   - If the package is exported privately, without qualification, then the
>     resource can be located by code in any module.
>
>   - If the package is exported privately to a specific set of modules
>     then the resource can be located by code in those modules.
>
>   - If the package is not exported privately in any way then the resource
>     cannot be located by code outside of the module itself.
>
> We revise the various `getResource*` methods to impose the above
> constraints:
>
>   - The `Module::getResourceAsStream` method allows a resource to be
>     located directly within a module, without first having to load a
>     class from that module.  This method is caller-sensitive.
>
>   - The `Class::getResource*` methods, when invoked upon a class defined
>     in a named module, only locate resources from within that module.
>     These methods are also caller-sensitive.
>
>   - The `ClassLoader::getResource*` methods will locate resources in
>     named modules subject to the above restrictions except that the
>     effective package of the resource must be exported privately without
>     qualification, since this method is not caller-sensitive.  Custom
>     class loaders that locate resources in modules should implement these
>     restrictions, but there is no way to force them to do so.  The order
>     of the elements of an enumeration returned by the `getResources`
>     method is unspecified, but for a given class loader `cl` the values
>     of `cl.getResources(name).nextElement()` and `cl.getResource(name)`
>     are always equal.
>
> Every package in an unnamed, automatic, or weak module is exported
> privately, so all of the resources in such modules can be located by
> code in all other modules.
>
> Notes
> -----
>
>   - The first proposal for these issues [5] suggested that we simply drop
>     the agreed resource-encapsulation requirement [2].  As suggested both
>     in the EG [6] and elsewhere [7][8], this revised proposal attempts to
>     strike a balance between practical compatibility and modular
>     encapsulation.
>
>   - This proposal leverages private exports, i.e., the `exports private`
>     directive, since the internal resources of a module are much like the
>     non-public elements of the types defined in a module.  This avoids
>     leaking resources inadvertently from packages that are exported
>     normally, i.e., without the `private` modifier.  It also avoids the
>     need to extend the syntax and semantics of module declarations with
>     a completely new directive just for resources, which seems like
>     overkill.
>
>
> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassFilesAsResources
> [2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#resource-encapsulation
> [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#ResourceEncapsulation
> [4] http://docs.oracle.com/javase/specs/jls/se8/html/jls-6.html#jls-PackageName
> [5] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000309.html
> [6] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000322.html
> [7] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-June/000050.html
> [8] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-June/000051.html
>

-- 
- DML

From mark.reinhold at oracle.com  Tue Sep 13 15:54:40 2016
From: mark.reinhold at oracle.com (Mark Reinhold)
Date: Tue, 13 Sep 2016 08:54:40 -0700 (PDT)
Subject: Proposed schedule change for JSR 376 (Java Platform Module System)
Message-ID: <20160913155440.68410CD3AF@eggemoggin.niobe.net>

I posted a number of proposals yesterday [1], including a redesign
of the package-export feature [2].  It's going to take more time than
remains in the previously-proposed schedule [3] for both us and the
wider community to evaluate those proposals and, also, to work through
the other issues that are already on our list [4].

I therefore suggest that we adjust our schedule as follows:

    Early Draft Review      January 2017
    Public Review           March 2017
    Proposed Final Draft    May 2017
    Final Release           July 2017

Please let me know by this time next week (16:00 UTC next Tuesday,
20 September) of any objections to this proposal.  (I'll propose a
corresponding change to the JSR 379 (Java SE 9) schedule shortly.)

- Mark


[1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000389.html
[2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000390.html
[3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-December/000196.html
[4] http://openjdk.java.net/projects/jigsaw/spec/issues/

From mark.reinhold at oracle.com  Tue Sep 13 16:12:31 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Tue, 13 Sep 2016 09:12:31 -0700
Subject: Artical on JPMS integration with existing module systems
In-Reply-To: <OFB9D82F52.23C5B2F8-ON80258027.0045F544-80258027.004CB66A@notes.na.collabserv.com>
References: <OF349B4469.ABDAB9AA-ON00258013.004B08BD-86258013.004B8BD5@notes.na.collabserv.com>
	<OFB9D82F52.23C5B2F8-ON80258027.0045F544-80258027.004CB66A@notes.na.collabserv.com>
Message-ID: <20160913091231.525832187eggemoggin.niobe.net>

2016/9/7 6:57:55 -0700, tim_ellison at uk.ibm.com:
> Obviously Tom has come at this integration challenge from his deep
> knowledge of a particular OSGi implementation.  There are some
> important observations, and minor tweaks to the current model that
> will improve JPMS.

Tom -- thanks very much for your detailed feedback.  I'll reply at
length in due course, but most likely not until after JavaOne.

- Mark

From mark.reinhold at oracle.com  Thu Sep 15 17:20:09 2016
From: mark.reinhold at oracle.com (mark.reinhold at oracle.com)
Date: Thu, 15 Sep 2016 10:20:09 -0700
Subject: Updated EA builds with initial implementations of recent proposals
Message-ID: <20160915102009.300350866eggemoggin.niobe.net>

FYI:

  http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009404.html

- Mark

From forax at univ-mlv.fr  Fri Sep 16 13:44:17 2016
From: forax at univ-mlv.fr (Remi Forax)
Date: Fri, 16 Sep 2016 15:44:17 +0200 (CEST)
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &
	#AwkwardStrongEncapsulation: Weak modules & private exports
In-Reply-To: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
Message-ID: <617353007.2424567.1474033457532.JavaMail.zimbra@u-pem.fr>

Hi all,
the goals of this proposal is
- to explicitly mark packages that allow 'deep' reflection,
- disable 'deep' reflection by default even for exported packages
- ease the transition by adding a new migration state between automatic module and strong module (weak module).

so 3 goals with one stone :)

Have i miss something obvious but the last goal, easing migration is not listed in the issue summary ?
While adding a new kind of module to ease migration is maybe a good idea, it also add a new kind of modules when thinking about the modules and their relationship, which seems a terrible idea. We are loosing the KISS principle of the current module system.

I fully agree that 'deep' reflection should be disallowed by default and can be allowed package by package.
But as Stephen Colebourne, i fail to see why enabling 'deep' reflection on a package is related to the export keyword. While 'deep' reflection is somewhat related to visibility, it's not only about visibility of a class/method/field only, by example you can change the value of a public final field of a public class using setAccessible. In my opinion, exposing the API to 'deep' reflection is a different concern as exporting a package, thus using 'private' as a modifier on export is a bad idea (it also doesn't work well with the notion of restricted export (export to ...)).
Adding a new keyword 'expose' as Stephen propose to enable 'deep' reflection is a better way to answer to the goal 1 and 2.  

About the other proposed items:
  - using transitive instead of public for require, i fully agree.
  - about removing the dynamic exports feature, a package declared as non exported but declared as exposed will have the same meaning, no ? if yes, i agree that the notion dynamic export should be removed.

regards,
R?mi

----- Mail original -----
> De: "Mark Reinhold" <mark.reinhold at oracle.com>
> ?: jpms-spec-experts at openjdk.java.net
> Envoy?: Lundi 12 Septembre 2016 17:08:01
> Objet: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &	#AwkwardStrongEncapsulation: Weak modules & private
> exports

> Issue summary
> -------------
> 
>  #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
>  libraries require reflective access to members of the non-exported
>  types of other modules; examples include dependency injection (Guice),
>  persistence (JPA), debugging tools, code-automation tools, and
>  serialization (XStream).  In some cases the particular library to be
>  used is not known until run time (e.g., Hibernate and EclipseLink both
>  implement JPA).  This capability is also sometimes used to work around
>  bugs in unchangeable code.  Access to non-exported packages can, at
>  present, only be done via command-line flags, which is extremely
>  awkward.  Provide an easier way for reflective code to access such
>  non-exported types. [1]
> 
>  #AwkwardStrongEncapsulation --- A non-public element of an exported
>  package can still be accessed via the `AccessibleObject::setAccessible`
>  method of the core reflection API.  The only way to strongly
>  encapsulate such an element is to move it to a non-exported package.
>  This makes it awkward, at best, to encapsulate the internals of a
>  package that defines a public API. [2]
> 
> Proposal
> --------
> 
> (Warning: This is somewhat long, and in the end it affects both `exports`
> and `requires` directives.)
> 
> Extend the language of module declarations with the concept of _weak
> modules_.  Weak modules make it easy to modularize components whose
> internals will be accessed by reflection-based frameworks.  Every
> package in a weak module has the following properties:
> 
>  (A) It is exported at both compile time and run time, as if by an
>      `exports` directive, and
> 
>  (B) Its non-public elements are available for _deep_ reflection, i.e.,
>      at run time they can be made accessible to code outside the module
>      via the `AccessibleObject::setAccessible` method of the core
>      reflection API.
> 
> In other words, every type defined in a weak module, whether public or
> not, is subject to exactly the same access checks as in Java SE 8 and
> earlier releases.
> 
> A weak module is declared by placing the modifier `weak` before the
> `module` keyword.  The declaration of a weak module cannot contain any
> explicit `exports` directives.  If the `weak` modifier does not appear
> before the `module` keyword then the declared module is _strong_, and
> it can contain explicit `exports` directives.
> 
> Suppose we have a module `foo.bar` which has an internal package
> `com.foo.bar.model` that contains entity classes to be manipulated by
> Hibernate, via core reflection.  Then the module declaration
> 
>    weak module foo.bar {
>        // No exports
>        requires hibernate.core;
>        requires hibernate.entitymanager;
>    }
> 
> exports the public types in `com.foo.bar.model`, and those of any other
> packages, in all phases.  It additionally makes all non-public elements
> of all packages available for deep reflection, enabling Hibernate to
> access such elements in the `com.foo.bar.model` package via the
> `setAccessible` method.
> 
> Weak modules simplify the process of migrating to modules.  The steps
> to convert an existing component into a module were, previously:
> 
>  (1) Make any changes necessary to get it working as an automatic
>      module (e.g., eliminate duplicate packages), and then
> 
>  (2) Write an explicit module declaration, which entails identifying
>      both the component's dependences (`requires`) and the packages
>      whose public types are to be made available to other modules
>      (`exports`).
> 
> With weak modules we can now divide the second step into two steps:
> 
>  (2a) Write an explicit module declaration for a weak module, which
>       entails identifying just the component's dependences (`requires`).
> 
>  (2b) Convert the weak module into a strong module, which entails
>       identifying the packages of the component whose public types
>       are to be made available to other modules (`exports`).
> 
> In other words, weak modules make it possible to focus first upon the
> reliable configuration of a module (`requires`), and then later think
> about its strong encapsulation (`exports`).
> 
> Weak modules are "weak" in what they export, but they remain subject
> to all of the constraints required to achieve reliable configuration.
> They do not read the unnamed module (i.e., the class path), they do not
> allow cycles in the module graph, and they do not allow split packages.
> Weak modules read named modules only as indicated by their `requires`
> directives, and they consume and provide services only as indicated by
> their `uses` and `provides` directives.
> 
>                                  * * *
> 
> In a strong module, an ordinary `exports` directive exports a package at
> both compile time and run time (property (A) above) but does not make its
> non-public types available for deep reflection (B).  In order to enable a
> package in a strong module to be exported in the same way as in a weak
> module we introduce the per-export modifier `private` to denote this
> second property.
> 
> If the above weak `foo.bar` module, e.g., contains some other packages
> besides `com.foo.bar.model`, and we wish to encapsulate those packages,
> we can convert it into a strong module with the declaration
> 
>    module foo.bar {
>        exports private com.foo.bar.model;
>        requires hibernate.core;
>        requires hibernate.entitymanager;
>    }
> 
> Now Hibernate can still access any public or non-public entity classes in
> the `com.foo.bar.model` package, but all the other packages are strongly
> encapsulated.
> 
> The `private` modifier should generally not be used to export a package
> containing an API, since normally an API's internal implementation
> details should be strongly encapsulated.  It may, however, be useful for
> legacy APIs whose internals are known to be accessed by existing code.
> 
> Every package in a weak module, an automatic module, or an unnamed module
> is exported as if by an `exports private` directive.
> 
> To ensure the integrity of the platform we expect few, if any, packages
> in the JDK itself to be exported with the `private` modifier.
> 
>                                  * * *
> 
> The new `private` modifier can also be used with qualified exports,
> though they interact with unqualified exports in a non-obvious way.
> 
>  - If you write `exports p` then you can also write `exports private
>    p to m`, so that code in module `m` can access the non-public types
>    of `p` via deep reflection but code outside of `m` can only access
>    the public types of `p`.
> 
>  - If you write `exports p to m1` then you can also write `exports
>    private p to m2`, so that code in `m2` can access the non-public
>    types of `p` via deep reflection, code in `m1` can access the
>    public types of `p`, but no code in any other module can access
>    any of the types of `p`.
> 
>  - If you write `exports private p` then you cannot also have a
>    qualified export of `p`, since code in all other modules already
>    has access to the non-public types of `p` via deep reflection.
> 
> Put informally, you can give your friends additional access, but you
> can't discriminate against them by giving them less access than everyone
> else.
> 
> As before, duplicate `exports` directives are not permitted, in order to
> ensure easy readability.  At most one `exports` directive is relevant to
> any given package/module pair, and it's easy to determine which one.
> 
>                                  * * *
> 
> The introduction of `private` as a modifier of `exports` directives calls
> the existing syntax of `requires public` even more strongly into question
> than before.  A module declaration of the form
> 
>    module foo.bar {
>        exports private com.foo.bar.baz;
>        requires public java.sql;
>    }
> 
> is likely to be very confusing to an uninformed reader.  The `private`
> modifier in the `exports` directive means that the private elements of
> the `com.foo.bar.baz` package are exported for deep reflection at run
> time.  The `public` modifier in the `requires` directive, however, does
> not mean that the public elements of the `java.sql` module are needed by
> this module; that is true of any plain `requires` directive.  It means
> that, additionally, any client of this module is granted implied
> readability to the `java.sql` module, thereby gaining access to all of
> its exported types.
> 
> To reduce this confusion we rename the `public` modifier in `requires`
> directives to `transitive`.  Thus the above example becomes
> 
>    module foo.bar {
>        exports private com.foo.bar.baz;
>        requires transitive java.sql;
>    }
> 
> This is potentially confusing in a different way, since in mathematics
> the term "transitive" is usually applied to an entire relation rather
> than to three specific elements of a set.  Its use here does not, in
> particular, mean that the resolver does not interpret plain `requires`
> directives when computing the transitive closure of a set of root
> modules.  "Transitive" as used here is in the more abstract sense,
> expressing the notion of conveying a property -- in this case, the
> readability of the required module -- from one thing to another.
> 
> 
> Notes
> -----
> 
>  - This is significantly different from the first proposal [3].  It adds
>    the notion of weak modules, to ease migration, and also the notion of
>    exporting a package without enabling deep reflection, to strengthen
>    encapsulation.
> 
>  - This proposal removes the notion of dynamic exports, which in the
>    presence of private exports would introduce considerable complexity
>    into the interactions between qualified and unqualified exports.
>    This means that it is no longer possible to export a package only at
>    run time, so it is no longer possible for the author of a module to
>    express the intent that the types of a non-API package are meant to
>    be available to frameworks for deep reflection at run time but
>    inaccessible at compile time.  The dynamic-export feature could, if
>    needed, be added in a future release.
> 
>  - A strong module with no exports makes no types accessible to code in
>    other modules while a weak module makes all of its types accessible,
>    both directly and via deep reflection.  The declarations of such
>    modules are, however, visually similar since most of their text lies
>    between the curly braces:
> 
>        module m1 {
>            requires ...;
>            uses ...;
>            provides ...;
>        }
> 
>        weak module m2 {
>            requires ...;
>            uses ...;
>            provides ...;
>        }
> 
>    We suspect that this visual similarity will not cause much confusion
>    in practice since strong modules that export no packages will be very
>    rare.
> 
>  - If a container is to ensure that a package in an application module
>    is available for deep reflection only by a trusted framework then it
>    can arrange for that by rewriting that module's descriptor, as
>    suggested previously [4], to insert the appropriate qualified private
>    export.  If there is a possibility that two modules will need to
>    export a package of the same name to the same framework module, as
>    suggested by Jason Greene [5], then the container should instead
>    inject a small class into each module whose static initializer
>    invokes the `Module::addExports` method in order to export the
>    package to the framework module.  There is no need any longer for
>    the resolution algorithm to take this scenario into account [6].
> 
>  - This proposal primarily addresses "friendly" uses of reflection, such
>    as dependency injection and persistence, in which the author of a
>    module knows in advance that one or more packages must be exported at
>    run time for deep reflective access by frameworks.  Intrusive access
>    to arbitrary packages of arbitrary modules by, e.g., serialization
>    frameworks or debugging tools, will still require the use of sharp
>    knives such as the `--add-exports` command-line option, the legacy
>    unsupported `sun.misc.Unsafe` API and related APIs, or JVM TI.
> 
>  - Using the `--add-exports` option or its equivalent remains awkward,
>    and sometimes it's the only way out.  To ease migration it's worth
>    considering some way for an application packaged as a JAR file to
>    include such options in its `MANIFEST.MF` file, as suggested by Simon
>    Nash [7].  This is tracked as #AddExportsInManifest [8].
> 
> 
> [1]
> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
> [2]
> http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
> [3]
> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
> [4] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
> [5] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
> [6] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
> [7] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
> [8] http://openjdk.java.net/projects/jigsaw/spec/issues/#AddExportsInManifest

From david.lloyd at redhat.com  Fri Sep 16 15:30:41 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Fri, 16 Sep 2016 10:30:41 -0500
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &
	#AwkwardStrongEncapsulation: Weak modules & private exports
In-Reply-To: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
Message-ID: <5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com>

After a fairly detailed review of this proposal, we have determined that 
it is not acceptable to Red Hat in its present form.  I will list the 
primary problems here, and then I'll start up discussion on jigsaw-dev 
of several possible solutions that could work for us.  I'll number these 
in case anyone wants to respond piece-wise.

#1) The "weak" designation appears to be pejorative

Under this solution, many existing frameworks and/or the modules which 
consume them must be marked "weak" in order to work correctly, or else a 
much more complex module descriptor must be used.  While this seems to 
be a stepping stone for migration, we find that it rather makes 
middleware appear as a second-class citizen.

#2) Weak modules are, in fact, weak

The idea of retaining the Java 8 rules for transition modules appears 
good on the surface, however the problem is that users may be forced to 
use weak modules indefinitely in order to interoperate with certain 
existing middleware and libraries, or else be faced with a potentially 
complex migration.  If the security benefits of "strong" modules are 
real, then users will be rightly inclined to move from weak modules to 
strong modules.  This can be severely problematic if existing 
middleware, libraries, and large applications cannot easily operate as 
or with "strong" modules or cannot do so in an "easy to learn"/"easy to 
use" fashion.  This also means that "weak" modules exist only for the 
purposes of maintaining a transition period - at some point they will 
inevitably transition from "convenience" to "attractive nuisance".  We 
believe that it is feasible to solve compatibility to a satisfactory 
degree without relegating users of existing software to a lower tier of 
security or functionality.

#3) Controlling reflection access is always bound up with controlling 
exports

The proposed exports system is simplistic but not exactly simple, yet 
several useful modes remain left unaccounted-for.  Internally we 
examined the usefulness of controlling access to public and so-called 
"deep" reflection in combination with whether or not a member was 
exported, and we came up with this logical table:

          +------------------+---------------------------+
          |                  |        Module is...       |
          |  Reflection is... +------------+--------------+
          |                  |  Exported  | Not Exported |
          +----------------------------------------------+
          |        Forbidden | Not Useful |    Useful    |
          +----------------------------------------------+
          |      Public Only |   Useful   |    Useful    |
          +----------------------------------------------+
          | Public & Private |   Useful   |    Useful    |
          +------------------+------------+--------------+

Some of these useful modes seem not to be available, or are not usable 
in an obvious or simple manner.  These modes are useful, possibly even 
necessary, to introduce in Java 9 in order to provide a good 
compatibility story with the many existing frameworks in the wild, in 
the absence of weak modules.

It has been observed that in some cases, the necessity of granting 
public or private reflective access to a module is actually bound not 
with what that module exports, but rather with what it requires or uses, 
which might be useful in consideration of a solution; for example, 
"using" a persistence framework which operates only on public types 
would imply that the module would grant back public-only reflective 
access to the framework which provides the service.

#4) There isn't always a container

The justification for not providing an easy mechanism to restrict 
private reflective access to a trusted framework is that a container can 
always rewrite a module or inject classes.  However there isn't always a 
container present - something which the existence of weak modules 
acknowledges.  Some frameworks and middleware are meant to be usable 
with or without a container, and it is undesirable to force these use 
cases into a weak module situation or to require a strong coupling in 
the module descriptor in this case.  Some other solution should be found 
that works both for containers and "regular" applications.

Broadly speaking, a container would be able to do additional things that 
are useful do to specifically in a container.  But a container should 
not be _necessary_ in order to do things that are generally useful, at 
least to the greatest extent possible.

#6) Not all reflection is 100% "friendly", and that's OK

Serialization frameworks generally need private reflective access to the 
module which contains the classes being serialized.  For some 
frameworks, this access will already be adequate.  Some existing 
frameworks utilize Unsafe to create uninitialized instances.  However 
other frameworks, including frameworks that seek to comply with the 
Serialization Specification [1], use ReflectionFactory, as that class 
provides (exclusively, as far as I know) the ability to acquire a 
constructor that produces a _partially_-initialized class where only the 
constructors of superclasses to a certain depth are called.  It is our 
view that it would be a regression to force such frameworks to use 
--add-exports or JVMTI, and using Unsafe for this purpose, or continuing 
to use it, (as I understand it) puts at risk the ability to perform 
optimizations which depend on entry point control, which I am given to 
understand are a strong motivator for many of the design constraints 
that exist on the Jigsaw implementation.  It should be possible for 
these frameworks to continue to use ReflectionFactory (unsupported 
though it may be) if suitably privileged (in the security manager 
sense), as long as all of the module(s) upon which the framework 
reflects have granted private reflective access to it (in the module 
declaration sense).  This may be imperfect in the event that class 
hierarchies are spread across several modules, but that is something 
that can be hashed out in detail on jigsaw-dev.

A better alternative to ReflectionFactory itself could be sought in 
future releases, if necessary, though one hopes that traditional 
serialization will die out in favor of more modern approaches, rendering 
the issue less relevant over time.  Admittedly, maybe a very long period 
of time.

#7) More loose coupling seems necessary and useful

In order for typical applications to function with modern middleware as 
modules, without compromising security, it may be necessary to enhance 
the loose coupling mechanism (uses/provides), or to provide an 
additional, similar mechanism, which allows a symbolic coupling which 
would allow modules to declare (in an abstract manner) modules which 
need to have reflective access to it in a "friendly" manner. 
Implementation ideas are forthcoming on jigsaw-dev.

Providing a way within the system to grant public and/or private 
reflection access in a specific manner, and doing so in a manner which 
is easy to use and not prejudicial against existing or future middleware 
and small or large applications which consume such middleware in the 
Java SE or EE space, is our most basic requirement for satisfactory 
resolution of this issue.  I will follow up on jigsaw-dev with some 
implementation thoughts and more specific requirements.

This list may not be exhaustive but it should be, at the least, a very 
good starting point for discussion, hopefully with a short path to 
implementation.

On 09/12/2016 10:08 AM, Mark Reinhold wrote:
> Issue summary
> -------------
>
>   #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
>   libraries require reflective access to members of the non-exported
>   types of other modules; examples include dependency injection (Guice),
>   persistence (JPA), debugging tools, code-automation tools, and
>   serialization (XStream).  In some cases the particular library to be
>   used is not known until run time (e.g., Hibernate and EclipseLink both
>   implement JPA).  This capability is also sometimes used to work around
>   bugs in unchangeable code.  Access to non-exported packages can, at
>   present, only be done via command-line flags, which is extremely
>   awkward.  Provide an easier way for reflective code to access such
>   non-exported types. [1]
>
>   #AwkwardStrongEncapsulation --- A non-public element of an exported
>   package can still be accessed via the `AccessibleObject::setAccessible`
>   method of the core reflection API.  The only way to strongly
>   encapsulate such an element is to move it to a non-exported package.
>   This makes it awkward, at best, to encapsulate the internals of a
>   package that defines a public API. [2]
>
> Proposal
> --------
>
> (Warning: This is somewhat long, and in the end it affects both `exports`
>  and `requires` directives.)
>
> Extend the language of module declarations with the concept of _weak
> modules_.  Weak modules make it easy to modularize components whose
> internals will be accessed by reflection-based frameworks.  Every
> package in a weak module has the following properties:
>
>   (A) It is exported at both compile time and run time, as if by an
>       `exports` directive, and
>
>   (B) Its non-public elements are available for _deep_ reflection, i.e.,
>       at run time they can be made accessible to code outside the module
>       via the `AccessibleObject::setAccessible` method of the core
>       reflection API.
>
> In other words, every type defined in a weak module, whether public or
> not, is subject to exactly the same access checks as in Java SE 8 and
> earlier releases.
>
> A weak module is declared by placing the modifier `weak` before the
> `module` keyword.  The declaration of a weak module cannot contain any
> explicit `exports` directives.  If the `weak` modifier does not appear
> before the `module` keyword then the declared module is _strong_, and
> it can contain explicit `exports` directives.
>
> Suppose we have a module `foo.bar` which has an internal package
> `com.foo.bar.model` that contains entity classes to be manipulated by
> Hibernate, via core reflection.  Then the module declaration
>
>     weak module foo.bar {
>         // No exports
>         requires hibernate.core;
>         requires hibernate.entitymanager;
>     }
>
> exports the public types in `com.foo.bar.model`, and those of any other
> packages, in all phases.  It additionally makes all non-public elements
> of all packages available for deep reflection, enabling Hibernate to
> access such elements in the `com.foo.bar.model` package via the
> `setAccessible` method.
>
> Weak modules simplify the process of migrating to modules.  The steps
> to convert an existing component into a module were, previously:
>
>   (1) Make any changes necessary to get it working as an automatic
>       module (e.g., eliminate duplicate packages), and then
>
>   (2) Write an explicit module declaration, which entails identifying
>       both the component's dependences (`requires`) and the packages
>       whose public types are to be made available to other modules
>       (`exports`).
>
> With weak modules we can now divide the second step into two steps:
>
>   (2a) Write an explicit module declaration for a weak module, which
>        entails identifying just the component's dependences (`requires`).
>
>   (2b) Convert the weak module into a strong module, which entails
>        identifying the packages of the component whose public types
>        are to be made available to other modules (`exports`).
>
> In other words, weak modules make it possible to focus first upon the
> reliable configuration of a module (`requires`), and then later think
> about its strong encapsulation (`exports`).
>
> Weak modules are "weak" in what they export, but they remain subject
> to all of the constraints required to achieve reliable configuration.
> They do not read the unnamed module (i.e., the class path), they do not
> allow cycles in the module graph, and they do not allow split packages.
> Weak modules read named modules only as indicated by their `requires`
> directives, and they consume and provide services only as indicated by
> their `uses` and `provides` directives.
>
>                                   * * *
>
> In a strong module, an ordinary `exports` directive exports a package at
> both compile time and run time (property (A) above) but does not make its
> non-public types available for deep reflection (B).  In order to enable a
> package in a strong module to be exported in the same way as in a weak
> module we introduce the per-export modifier `private` to denote this
> second property.
>
> If the above weak `foo.bar` module, e.g., contains some other packages
> besides `com.foo.bar.model`, and we wish to encapsulate those packages,
> we can convert it into a strong module with the declaration
>
>     module foo.bar {
>         exports private com.foo.bar.model;
>         requires hibernate.core;
>         requires hibernate.entitymanager;
>     }
>
> Now Hibernate can still access any public or non-public entity classes in
> the `com.foo.bar.model` package, but all the other packages are strongly
> encapsulated.
>
> The `private` modifier should generally not be used to export a package
> containing an API, since normally an API's internal implementation
> details should be strongly encapsulated.  It may, however, be useful for
> legacy APIs whose internals are known to be accessed by existing code.
>
> Every package in a weak module, an automatic module, or an unnamed module
> is exported as if by an `exports private` directive.
>
> To ensure the integrity of the platform we expect few, if any, packages
> in the JDK itself to be exported with the `private` modifier.
>
>                                   * * *
>
> The new `private` modifier can also be used with qualified exports,
> though they interact with unqualified exports in a non-obvious way.
>
>   - If you write `exports p` then you can also write `exports private
>     p to m`, so that code in module `m` can access the non-public types
>     of `p` via deep reflection but code outside of `m` can only access
>     the public types of `p`.
>
>   - If you write `exports p to m1` then you can also write `exports
>     private p to m2`, so that code in `m2` can access the non-public
>     types of `p` via deep reflection, code in `m1` can access the
>     public types of `p`, but no code in any other module can access
>     any of the types of `p`.
>
>   - If you write `exports private p` then you cannot also have a
>     qualified export of `p`, since code in all other modules already
>     has access to the non-public types of `p` via deep reflection.
>
> Put informally, you can give your friends additional access, but you
> can't discriminate against them by giving them less access than everyone
> else.
>
> As before, duplicate `exports` directives are not permitted, in order to
> ensure easy readability.  At most one `exports` directive is relevant to
> any given package/module pair, and it's easy to determine which one.
>
>                                   * * *
>
> The introduction of `private` as a modifier of `exports` directives calls
> the existing syntax of `requires public` even more strongly into question
> than before.  A module declaration of the form
>
>     module foo.bar {
>         exports private com.foo.bar.baz;
>         requires public java.sql;
>     }
>
> is likely to be very confusing to an uninformed reader.  The `private`
> modifier in the `exports` directive means that the private elements of
> the `com.foo.bar.baz` package are exported for deep reflection at run
> time.  The `public` modifier in the `requires` directive, however, does
> not mean that the public elements of the `java.sql` module are needed by
> this module; that is true of any plain `requires` directive.  It means
> that, additionally, any client of this module is granted implied
> readability to the `java.sql` module, thereby gaining access to all of
> its exported types.
>
> To reduce this confusion we rename the `public` modifier in `requires`
> directives to `transitive`.  Thus the above example becomes
>
>     module foo.bar {
>         exports private com.foo.bar.baz;
>         requires transitive java.sql;
>     }
>
> This is potentially confusing in a different way, since in mathematics
> the term "transitive" is usually applied to an entire relation rather
> than to three specific elements of a set.  Its use here does not, in
> particular, mean that the resolver does not interpret plain `requires`
> directives when computing the transitive closure of a set of root
> modules.  "Transitive" as used here is in the more abstract sense,
> expressing the notion of conveying a property -- in this case, the
> readability of the required module -- from one thing to another.
>
>
> Notes
> -----
>
>   - This is significantly different from the first proposal [3].  It adds
>     the notion of weak modules, to ease migration, and also the notion of
>     exporting a package without enabling deep reflection, to strengthen
>     encapsulation.
>
>   - This proposal removes the notion of dynamic exports, which in the
>     presence of private exports would introduce considerable complexity
>     into the interactions between qualified and unqualified exports.
>     This means that it is no longer possible to export a package only at
>     run time, so it is no longer possible for the author of a module to
>     express the intent that the types of a non-API package are meant to
>     be available to frameworks for deep reflection at run time but
>     inaccessible at compile time.  The dynamic-export feature could, if
>     needed, be added in a future release.
>
>   - A strong module with no exports makes no types accessible to code in
>     other modules while a weak module makes all of its types accessible,
>     both directly and via deep reflection.  The declarations of such
>     modules are, however, visually similar since most of their text lies
>     between the curly braces:
>
>         module m1 {
>             requires ...;
>             uses ...;
>             provides ...;
>         }
>
>         weak module m2 {
>             requires ...;
>             uses ...;
>             provides ...;
>         }
>
>     We suspect that this visual similarity will not cause much confusion
>     in practice since strong modules that export no packages will be very
>     rare.
>
>   - If a container is to ensure that a package in an application module
>     is available for deep reflection only by a trusted framework then it
>     can arrange for that by rewriting that module's descriptor, as
>     suggested previously [4], to insert the appropriate qualified private
>     export.  If there is a possibility that two modules will need to
>     export a package of the same name to the same framework module, as
>     suggested by Jason Greene [5], then the container should instead
>     inject a small class into each module whose static initializer
>     invokes the `Module::addExports` method in order to export the
>     package to the framework module.  There is no need any longer for
>     the resolution algorithm to take this scenario into account [6].
>
>   - This proposal primarily addresses "friendly" uses of reflection, such
>     as dependency injection and persistence, in which the author of a
>     module knows in advance that one or more packages must be exported at
>     run time for deep reflective access by frameworks.  Intrusive access
>     to arbitrary packages of arbitrary modules by, e.g., serialization
>     frameworks or debugging tools, will still require the use of sharp
>     knives such as the `--add-exports` command-line option, the legacy
>     unsupported `sun.misc.Unsafe` API and related APIs, or JVM TI.
>
>   - Using the `--add-exports` option or its equivalent remains awkward,
>     and sometimes it's the only way out.  To ease migration it's worth
>     considering some way for an application packaged as a JAR file to
>     include such options in its `MANIFEST.MF` file, as suggested by Simon
>     Nash [7].  This is tracked as #AddExportsInManifest [8].
>
>
> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
> [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
> [3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
> [4] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
> [5] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
> [6] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
> [7] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
> [8] http://openjdk.java.net/projects/jigsaw/spec/issues/#AddExportsInManifest
>

-- 
- DML

From david.lloyd at redhat.com  Fri Sep 16 17:51:47 2016
From: david.lloyd at redhat.com (David M. Lloyd)
Date: Fri, 16 Sep 2016 12:51:47 -0500
Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) &
	#AwkwardStrongEncapsulation: Weak modules & private exports
In-Reply-To: <5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com>
References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net>
	<5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com>
Message-ID: <fa5bc961-ae7f-1451-6e8d-ff6ae9999155@redhat.com>

[1] 
https://docs.oracle.com/javase/8/docs/platform/serialization/spec/serialTOC.html

On 09/16/2016 10:30 AM, David M. Lloyd wrote:
> After a fairly detailed review of this proposal, we have determined that
> it is not acceptable to Red Hat in its present form.  I will list the
> primary problems here, and then I'll start up discussion on jigsaw-dev
> of several possible solutions that could work for us.  I'll number these
> in case anyone wants to respond piece-wise.
>
> #1) The "weak" designation appears to be pejorative
>
> Under this solution, many existing frameworks and/or the modules which
> consume them must be marked "weak" in order to work correctly, or else a
> much more complex module descriptor must be used.  While this seems to
> be a stepping stone for migration, we find that it rather makes
> middleware appear as a second-class citizen.
>
> #2) Weak modules are, in fact, weak
>
> The idea of retaining the Java 8 rules for transition modules appears
> good on the surface, however the problem is that users may be forced to
> use weak modules indefinitely in order to interoperate with certain
> existing middleware and libraries, or else be faced with a potentially
> complex migration.  If the security benefits of "strong" modules are
> real, then users will be rightly inclined to move from weak modules to
> strong modules.  This can be severely problematic if existing
> middleware, libraries, and large applications cannot easily operate as
> or with "strong" modules or cannot do so in an "easy to learn"/"easy to
> use" fashion.  This also means that "weak" modules exist only for the
> purposes of maintaining a transition period - at some point they will
> inevitably transition from "convenience" to "attractive nuisance".  We
> believe that it is feasible to solve compatibility to a satisfactory
> degree without relegating users of existing software to a lower tier of
> security or functionality.
>
> #3) Controlling reflection access is always bound up with controlling
> exports
>
> The proposed exports system is simplistic but not exactly simple, yet
> several useful modes remain left unaccounted-for.  Internally we
> examined the usefulness of controlling access to public and so-called
> "deep" reflection in combination with whether or not a member was
> exported, and we came up with this logical table:
>
>          +------------------+---------------------------+
>          |                  |        Module is...       |
>          |  Reflection is... +------------+--------------+
>          |                  |  Exported  | Not Exported |
>          +----------------------------------------------+
>          |        Forbidden | Not Useful |    Useful    |
>          +----------------------------------------------+
>          |      Public Only |   Useful   |    Useful    |
>          +----------------------------------------------+
>          | Public & Private |   Useful   |    Useful    |
>          +------------------+------------+--------------+
>
> Some of these useful modes seem not to be available, or are not usable
> in an obvious or simple manner.  These modes are useful, possibly even
> necessary, to introduce in Java 9 in order to provide a good
> compatibility story with the many existing frameworks in the wild, in
> the absence of weak modules.
>
> It has been observed that in some cases, the necessity of granting
> public or private reflective access to a module is actually bound not
> with what that module exports, but rather with what it requires or uses,
> which might be useful in consideration of a solution; for example,
> "using" a persistence framework which operates only on public types
> would imply that the module would grant back public-only reflective
> access to the framework which provides the service.
>
> #4) There isn't always a container
>
> The justification for not providing an easy mechanism to restrict
> private reflective access to a trusted framework is that a container can
> always rewrite a module or inject classes.  However there isn't always a
> container present - something which the existence of weak modules
> acknowledges.  Some frameworks and middleware are meant to be usable
> with or without a container, and it is undesirable to force these use
> cases into a weak module situation or to require a strong coupling in
> the module descriptor in this case.  Some other solution should be found
> that works both for containers and "regular" applications.
>
> Broadly speaking, a container would be able to do additional things that
> are useful do to specifically in a container.  But a container should
> not be _necessary_ in order to do things that are generally useful, at
> least to the greatest extent possible.
>
> #6) Not all reflection is 100% "friendly", and that's OK
>
> Serialization frameworks generally need private reflective access to the
> module which contains the classes being serialized.  For some
> frameworks, this access will already be adequate.  Some existing
> frameworks utilize Unsafe to create uninitialized instances.  However
> other frameworks, including frameworks that seek to comply with the
> Serialization Specification [1], use ReflectionFactory, as that class
> provides (exclusively, as far as I know) the ability to acquire a
> constructor that produces a _partially_-initialized class where only the
> constructors of superclasses to a certain depth are called.  It is our
> view that it would be a regression to force such frameworks to use
> --add-exports or JVMTI, and using Unsafe for this purpose, or continuing
> to use it, (as I understand it) puts at risk the ability to perform
> optimizations which depend on entry point control, which I am given to
> understand are a strong motivator for many of the design constraints
> that exist on the Jigsaw implementation.  It should be possible for
> these frameworks to continue to use ReflectionFactory (unsupported
> though it may be) if suitably privileged (in the security manager
> sense), as long as all of the module(s) upon which the framework
> reflects have granted private reflective access to it (in the module
> declaration sense).  This may be imperfect in the event that class
> hierarchies are spread across several modules, but that is something
> that can be hashed out in detail on jigsaw-dev.
>
> A better alternative to ReflectionFactory itself could be sought in
> future releases, if necessary, though one hopes that traditional
> serialization will die out in favor of more modern approaches, rendering
> the issue less relevant over time.  Admittedly, maybe a very long period
> of time.
>
> #7) More loose coupling seems necessary and useful
>
> In order for typical applications to function with modern middleware as
> modules, without compromising security, it may be necessary to enhance
> the loose coupling mechanism (uses/provides), or to provide an
> additional, similar mechanism, which allows a symbolic coupling which
> would allow modules to declare (in an abstract manner) modules which
> need to have reflective access to it in a "friendly" manner.
> Implementation ideas are forthcoming on jigsaw-dev.
>
> Providing a way within the system to grant public and/or private
> reflection access in a specific manner, and doing so in a manner which
> is easy to use and not prejudicial against existing or future middleware
> and small or large applications which consume such middleware in the
> Java SE or EE space, is our most basic requirement for satisfactory
> resolution of this issue.  I will follow up on jigsaw-dev with some
> implementation thoughts and more specific requirements.
>
> This list may not be exhaustive but it should be, at the least, a very
> good starting point for discussion, hopefully with a short path to
> implementation.
>
> On 09/12/2016 10:08 AM, Mark Reinhold wrote:
>> Issue summary
>> -------------
>>
>>   #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
>>   libraries require reflective access to members of the non-exported
>>   types of other modules; examples include dependency injection (Guice),
>>   persistence (JPA), debugging tools, code-automation tools, and
>>   serialization (XStream).  In some cases the particular library to be
>>   used is not known until run time (e.g., Hibernate and EclipseLink both
>>   implement JPA).  This capability is also sometimes used to work around
>>   bugs in unchangeable code.  Access to non-exported packages can, at
>>   present, only be done via command-line flags, which is extremely
>>   awkward.  Provide an easier way for reflective code to access such
>>   non-exported types. [1]
>>
>>   #AwkwardStrongEncapsulation --- A non-public element of an exported
>>   package can still be accessed via the `AccessibleObject::setAccessible`
>>   method of the core reflection API.  The only way to strongly
>>   encapsulate such an element is to move it to a non-exported package.
>>   This makes it awkward, at best, to encapsulate the internals of a
>>   package that defines a public API. [2]
>>
>> Proposal
>> --------
>>
>> (Warning: This is somewhat long, and in the end it affects both `exports`
>>  and `requires` directives.)
>>
>> Extend the language of module declarations with the concept of _weak
>> modules_.  Weak modules make it easy to modularize components whose
>> internals will be accessed by reflection-based frameworks.  Every
>> package in a weak module has the following properties:
>>
>>   (A) It is exported at both compile time and run time, as if by an
>>       `exports` directive, and
>>
>>   (B) Its non-public elements are available for _deep_ reflection, i.e.,
>>       at run time they can be made accessible to code outside the module
>>       via the `AccessibleObject::setAccessible` method of the core
>>       reflection API.
>>
>> In other words, every type defined in a weak module, whether public or
>> not, is subject to exactly the same access checks as in Java SE 8 and
>> earlier releases.
>>
>> A weak module is declared by placing the modifier `weak` before the
>> `module` keyword.  The declaration of a weak module cannot contain any
>> explicit `exports` directives.  If the `weak` modifier does not appear
>> before the `module` keyword then the declared module is _strong_, and
>> it can contain explicit `exports` directives.
>>
>> Suppose we have a module `foo.bar` which has an internal package
>> `com.foo.bar.model` that contains entity classes to be manipulated by
>> Hibernate, via core reflection.  Then the module declaration
>>
>>     weak module foo.bar {
>>         // No exports
>>         requires hibernate.core;
>>         requires hibernate.entitymanager;
>>     }
>>
>> exports the public types in `com.foo.bar.model`, and those of any other
>> packages, in all phases.  It additionally makes all non-public elements
>> of all packages available for deep reflection, enabling Hibernate to
>> access such elements in the `com.foo.bar.model` package via the
>> `setAccessible` method.
>>
>> Weak modules simplify the process of migrating to modules.  The steps
>> to convert an existing component into a module were, previously:
>>
>>   (1) Make any changes necessary to get it working as an automatic
>>       module (e.g., eliminate duplicate packages), and then
>>
>>   (2) Write an explicit module declaration, which entails identifying
>>       both the component's dependences (`requires`) and the packages
>>       whose public types are to be made available to other modules
>>       (`exports`).
>>
>> With weak modules we can now divide the second step into two steps:
>>
>>   (2a) Write an explicit module declaration for a weak module, which
>>        entails identifying just the component's dependences (`requires`).
>>
>>   (2b) Convert the weak module into a strong module, which entails
>>        identifying the packages of the component whose public types
>>        are to be made available to other modules (`exports`).
>>
>> In other words, weak modules make it possible to focus first upon the
>> reliable configuration of a module (`requires`), and then later think
>> about its strong encapsulation (`exports`).
>>
>> Weak modules are "weak" in what they export, but they remain subject
>> to all of the constraints required to achieve reliable configuration.
>> They do not read the unnamed module (i.e., the class path), they do not
>> allow cycles in the module graph, and they do not allow split packages.
>> Weak modules read named modules only as indicated by their `requires`
>> directives, and they consume and provide services only as indicated by
>> their `uses` and `provides` directives.
>>
>>                                   * * *
>>
>> In a strong module, an ordinary `exports` directive exports a package at
>> both compile time and run time (property (A) above) but does not make its
>> non-public types available for deep reflection (B).  In order to enable a
>> package in a strong module to be exported in the same way as in a weak
>> module we introduce the per-export modifier `private` to denote this
>> second property.
>>
>> If the above weak `foo.bar` module, e.g., contains some other packages
>> besides `com.foo.bar.model`, and we wish to encapsulate those packages,
>> we can convert it into a strong module with the declaration
>>
>>     module foo.bar {
>>         exports private com.foo.bar.model;
>>         requires hibernate.core;
>>         requires hibernate.entitymanager;
>>     }
>>
>> Now Hibernate can still access any public or non-public entity classes in
>> the `com.foo.bar.model` package, but all the other packages are strongly
>> encapsulated.
>>
>> The `private` modifier should generally not be used to export a package
>> containing an API, since normally an API's internal implementation
>> details should be strongly encapsulated.  It may, however, be useful for
>> legacy APIs whose internals are known to be accessed by existing code.
>>
>> Every package in a weak module, an automatic module, or an unnamed module
>> is exported as if by an `exports private` directive.
>>
>> To ensure the integrity of the platform we expect few, if any, packages
>> in the JDK itself to be exported with the `private` modifier.
>>
>>                                   * * *
>>
>> The new `private` modifier can also be used with qualified exports,
>> though they interact with unqualified exports in a non-obvious way.
>>
>>   - If you write `exports p` then you can also write `exports private
>>     p to m`, so that code in module `m` can access the non-public types
>>     of `p` via deep reflection but code outside of `m` can only access
>>     the public types of `p`.
>>
>>   - If you write `exports p to m1` then you can also write `exports
>>     private p to m2`, so that code in `m2` can access the non-public
>>     types of `p` via deep reflection, code in `m1` can access the
>>     public types of `p`, but no code in any other module can access
>>     any of the types of `p`.
>>
>>   - If you write `exports private p` then you cannot also have a
>>     qualified export of `p`, since code in all other modules already
>>     has access to the non-public types of `p` via deep reflection.
>>
>> Put informally, you can give your friends additional access, but you
>> can't discriminate against them by giving them less access than everyone
>> else.
>>
>> As before, duplicate `exports` directives are not permitted, in order to
>> ensure easy readability.  At most one `exports` directive is relevant to
>> any given package/module pair, and it's easy to determine which one.
>>
>>                                   * * *
>>
>> The introduction of `private` as a modifier of `exports` directives calls
>> the existing syntax of `requires public` even more strongly into question
>> than before.  A module declaration of the form
>>
>>     module foo.bar {
>>         exports private com.foo.bar.baz;
>>         requires public java.sql;
>>     }
>>
>> is likely to be very confusing to an uninformed reader.  The `private`
>> modifier in the `exports` directive means that the private elements of
>> the `com.foo.bar.baz` package are exported for deep reflection at run
>> time.  The `public` modifier in the `requires` directive, however, does
>> not mean that the public elements of the `java.sql` module are needed by
>> this module; that is true of any plain `requires` directive.  It means
>> that, additionally, any client of this module is granted implied
>> readability to the `java.sql` module, thereby gaining access to all of
>> its exported types.
>>
>> To reduce this confusion we rename the `public` modifier in `requires`
>> directives to `transitive`.  Thus the above example becomes
>>
>>     module foo.bar {
>>         exports private com.foo.bar.baz;
>>         requires transitive java.sql;
>>     }
>>
>> This is potentially confusing in a different way, since in mathematics
>> the term "transitive" is usually applied to an entire relation rather
>> than to three specific elements of a set.  Its use here does not, in
>> particular, mean that the resolver does not interpret plain `requires`
>> directives when computing the transitive closure of a set of root
>> modules.  "Transitive" as used here is in the more abstract sense,
>> expressing the notion of conveying a property -- in this case, the
>> readability of the required module -- from one thing to another.
>>
>>
>> Notes
>> -----
>>
>>   - This is significantly different from the first proposal [3].  It adds
>>     the notion of weak modules, to ease migration, and also the notion of
>>     exporting a package without enabling deep reflection, to strengthen
>>     encapsulation.
>>
>>   - This proposal removes the notion of dynamic exports, which in the
>>     presence of private exports would introduce considerable complexity
>>     into the interactions between qualified and unqualified exports.
>>     This means that it is no longer possible to export a package only at
>>     run time, so it is no longer possible for the author of a module to
>>     express the intent that the types of a non-API package are meant to
>>     be available to frameworks for deep reflection at run time but
>>     inaccessible at compile time.  The dynamic-export feature could, if
>>     needed, be added in a future release.
>>
>>   - A strong module with no exports makes no types accessible to code in
>>     other modules while a weak module makes all of its types accessible,
>>     both directly and via deep reflection.  The declarations of such
>>     modules are, however, visually similar since most of their text lies
>>     between the curly braces:
>>
>>         module m1 {
>>             requires ...;
>>             uses ...;
>>             provides ...;
>>         }
>>
>>         weak module m2 {
>>             requires ...;
>>             uses ...;
>>             provides ...;
>>         }
>>
>>     We suspect that this visual similarity will not cause much confusion
>>     in practice since strong modules that export no packages will be very
>>     rare.
>>
>>   - If a container is to ensure that a package in an application module
>>     is available for deep reflection only by a trusted framework then it
>>     can arrange for that by rewriting that module's descriptor, as
>>     suggested previously [4], to insert the appropriate qualified private
>>     export.  If there is a possibility that two modules will need to
>>     export a package of the same name to the same framework module, as
>>     suggested by Jason Greene [5], then the container should instead
>>     inject a small class into each module whose static initializer
>>     invokes the `Module::addExports` method in order to export the
>>     package to the framework module.  There is no need any longer for
>>     the resolution algorithm to take this scenario into account [6].
>>
>>   - This proposal primarily addresses "friendly" uses of reflection, such
>>     as dependency injection and persistence, in which the author of a
>>     module knows in advance that one or more packages must be exported at
>>     run time for deep reflective access by frameworks.  Intrusive access
>>     to arbitrary packages of arbitrary modules by, e.g., serialization
>>     frameworks or debugging tools, will still require the use of sharp
>>     knives such as the `--add-exports` command-line option, the legacy
>>     unsupported `sun.misc.Unsafe` API and related APIs, or JVM TI.
>>
>>   - Using the `--add-exports` option or its equivalent remains awkward,
>>     and sometimes it's the only way out.  To ease migration it's worth
>>     considering some way for an application packaged as a JAR file to
>>     include such options in its `MANIFEST.MF` file, as suggested by Simon
>>     Nash [7].  This is tracked as #AddExportsInManifest [8].
>>
>>
>> [1]
>> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
>>
>> [2]
>> http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
>>
>> [3]
>> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
>>
>> [4]
>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
>> [5]
>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
>> [6]
>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
>> [7]
>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
>>
>> [8]
>> http://openjdk.java.net/projects/jigsaw/spec/issues/#AddExportsInManifest
>>
>

-- 
- DML