Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports

[Mark Reinhold]

Issue summary
-------------

  #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
  libraries require reflective access to members of the non-exported
  types of other modules; examples include dependency injection (Guice),
  persistence (JPA), debugging tools, code-automation tools, and
  serialization (XStream).  In some cases the particular library to be
  used is not known until run time (e.g., Hibernate and EclipseLink both
  implement JPA).  This capability is also sometimes used to work around
  bugs in unchangeable code.  Access to non-exported packages can, at
  present, only be done via command-line flags, which is extremely
  awkward.  Provide an easier way for reflective code to access such
  non-exported types. [1]

  #AwkwardStrongEncapsulation --- A non-public element of an exported
  package can still be accessed via the `AccessibleObject::setAccessible`
  method of the core reflection API.  The only way to strongly
  encapsulate such an element is to move it to a non-exported package.
  This makes it awkward, at best, to encapsulate the internals of a
  package that defines a public API. [2]

Proposal
--------

(Warning: This is somewhat long, and in the end it affects both `exports`
 and `requires` directives.)

Extend the language of module declarations with the concept of _weak
modules_.  Weak modules make it easy to modularize components whose
internals will be accessed by reflection-based frameworks.  Every
package in a weak module has the following properties:

  (A) It is exported at both compile time and run time, as if by an
      `exports` directive, and

  (B) Its non-public elements are available for _deep_ reflection, i.e.,
      at run time they can be made accessible to code outside the module
      via the `AccessibleObject::setAccessible` method of the core
      reflection API.

In other words, every type defined in a weak module, whether public or
not, is subject to exactly the same access checks as in Java SE 8 and
earlier releases.

A weak module is declared by placing the modifier `weak` before the
`module` keyword.  The declaration of a weak module cannot contain any
explicit `exports` directives.  If the `weak` modifier does not appear
before the `module` keyword then the declared module is _strong_, and
it can contain explicit `exports` directives.

Suppose we have a module `foo.bar` which has an internal package
`com.foo.bar.model` that contains entity classes to be manipulated by
Hibernate, via core reflection.  Then the module declaration

    weak module foo.bar {
        // No exports
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

exports the public types in `com.foo.bar.model`, and those of any other
packages, in all phases.  It additionally makes all non-public elements
of all packages available for deep reflection, enabling Hibernate to
access such elements in the `com.foo.bar.model` package via the
`setAccessible` method.

Weak modules simplify the process of migrating to modules.  The steps
to convert an existing component into a module were, previously:

  (1) Make any changes necessary to get it working as an automatic
      module (e.g., eliminate duplicate packages), and then

  (2) Write an explicit module declaration, which entails identifying
      both the component's dependences (`requires`) and the packages
      whose public types are to be made available to other modules
      (`exports`).

With weak modules we can now divide the second step into two steps:

  (2a) Write an explicit module declaration for a weak module, which
       entails identifying just the component's dependences (`requires`).

  (2b) Convert the weak module into a strong module, which entails
       identifying the packages of the component whose public types
       are to be made available to other modules (`exports`).

In other words, weak modules make it possible to focus first upon the
reliable configuration of a module (`requires`), and then later think
about its strong encapsulation (`exports`).

Weak modules are "weak" in what they export, but they remain subject
to all of the constraints required to achieve reliable configuration.
They do not read the unnamed module (i.e., the class path), they do not
allow cycles in the module graph, and they do not allow split packages.
Weak modules read named modules only as indicated by their `requires`
directives, and they consume and provide services only as indicated by
their `uses` and `provides` directives.

                                  * * *

In a strong module, an ordinary `exports` directive exports a package at
both compile time and run time (property (A) above) but does not make its
non-public types available for deep reflection (B).  In order to enable a
package in a strong module to be exported in the same way as in a weak
module we introduce the per-export modifier `private` to denote this
second property.

If the above weak `foo.bar` module, e.g., contains some other packages
besides `com.foo.bar.model`, and we wish to encapsulate those packages,
we can convert it into a strong module with the declaration

    module foo.bar {
        exports private com.foo.bar.model;
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

Now Hibernate can still access any public or non-public entity classes in
the `com.foo.bar.model` package, but all the other packages are strongly
encapsulated.

The `private` modifier should generally not be used to export a package
containing an API, since normally an API's internal implementation
details should be strongly encapsulated.  It may, however, be useful for
legacy APIs whose internals are known to be accessed by existing code.

Every package in a weak module, an automatic module, or an unnamed module
is exported as if by an `exports private` directive.

To ensure the integrity of the platform we expect few, if any, packages
in the JDK itself to be exported with the `private` modifier.

                                  * * *

The new `private` modifier can also be used with qualified exports,
though they interact with unqualified exports in a non-obvious way.

  - If you write `exports p` then you can also write `exports private
    p to m`, so that code in module `m` can access the non-public types
    of `p` via deep reflection but code outside of `m` can only access
    the public types of `p`.

  - If you write `exports p to m1` then you can also write `exports
    private p to m2`, so that code in `m2` can access the non-public
    types of `p` via deep reflection, code in `m1` can access the
    public types of `p`, but no code in any other module can access
    any of the types of `p`.

  - If you write `exports private p` then you cannot also have a
    qualified export of `p`, since code in all other modules already
    has access to the non-public types of `p` via deep reflection.

Put informally, you can give your friends additional access, but you
can't discriminate against them by giving them less access than everyone
else.

As before, duplicate `exports` directives are not permitted, in order to
ensure easy readability.  At most one `exports` directive is relevant to
any given package/module pair, and it's easy to determine which one.

                                  * * *

The introduction of `private` as a modifier of `exports` directives calls
the existing syntax of `requires public` even more strongly into question
than before.  A module declaration of the form

    module foo.bar {
        exports private com.foo.bar.baz;
        requires public java.sql;
    }

is likely to be very confusing to an uninformed reader.  The `private`
modifier in the `exports` directive means that the private elements of
the `com.foo.bar.baz` package are exported for deep reflection at run
time.  The `public` modifier in the `requires` directive, however, does
not mean that the public elements of the `java.sql` module are needed by
this module; that is true of any plain `requires` directive.  It means
that, additionally, any client of this module is granted implied
readability to the `java.sql` module, thereby gaining access to all of
its exported types.

To reduce this confusion we rename the `public` modifier in `requires`
directives to `transitive`.  Thus the above example becomes

    module foo.bar {
        exports private com.foo.bar.baz;
        requires transitive java.sql;
    }

This is potentially confusing in a different way, since in mathematics
the term "transitive" is usually applied to an entire relation rather
than to three specific elements of a set.  Its use here does not, in
particular, mean that the resolver does not interpret plain `requires`
directives when computing the transitive closure of a set of root
modules.  "Transitive" as used here is in the more abstract sense,
expressing the notion of conveying a property -- in this case, the
readability of the required module -- from one thing to another.


Notes
-----

  - This is significantly different from the first proposal [3].  It adds
    the notion of weak modules, to ease migration, and also the notion of
    exporting a package without enabling deep reflection, to strengthen
    encapsulation.

  - This proposal removes the notion of dynamic exports, which in the
    presence of private exports would introduce considerable complexity
    into the interactions between qualified and unqualified exports.
    This means that it is no longer possible to export a package only at
    run time, so it is no longer possible for the author of a module to
    express the intent that the types of a non-API package are meant to
    be available to frameworks for deep reflection at run time but
    inaccessible at compile time.  The dynamic-export feature could, if
    needed, be added in a future release.

  - A strong module with no exports makes no types accessible to code in
    other modules while a weak module makes all of its types accessible,
    both directly and via deep reflection.  The declarations of such
    modules are, however, visually similar since most of their text lies
    between the curly braces:

        module m1 {
            requires ...;
            uses ...;
            provides ...;
        }

        weak module m2 {
            requires ...;
            uses ...;
            provides ...;
        }

    We suspect that this visual similarity will not cause much confusion
    in practice since strong modules that export no packages will be very
    rare.

  - If a container is to ensure that a package in an application module
    is available for deep reflection only by a trusted framework then it
    can arrange for that by rewriting that module's descriptor, as
    suggested previously [4], to insert the appropriate qualified private
    export.  If there is a possibility that two modules will need to
    export a package of the same name to the same framework module, as
    suggested by Jason Greene [5], then the container should instead
    inject a small class into each module whose static initializer
    invokes the `Module::addExports` method in order to export the
    package to the framework module.  There is no need any longer for
    the resolution algorithm to take this scenario into account [6].

  - This proposal primarily addresses "friendly" uses of reflection, such
    as dependency injection and persistence, in which the author of a
    module knows in advance that one or more packages must be exported at
    run time for deep reflective access by frameworks.  Intrusive access
    to arbitrary packages of arbitrary modules by, e.g., serialization
    frameworks or debugging tools, will still require the use of sharp
    knives such as the `--add-exports` command-line option, the legacy
    unsupported `sun.misc.Unsafe` API and related APIs, or JVM TI.

  - Using the `--add-exports` option or its equivalent remains awkward,
    and sometimes it's the only way out.  To ease migration it's worth
    considering some way for an application packaged as a JAR file to
    include such options in its `MANIFEST.MF` file, as suggested by Simon
    Nash [7].  This is tracked as #AddExportsInManifest [8].


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
[2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
[3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
[4] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
[5] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
[6] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
[7] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
[8] http://openjdk.java.net/projects/jigsaw/spec/issues/#AddExportsInManifest