Proposal (revised^2): #ReflectiveAccessToNonExportedTypes & #AwkwardStrongEncapsulation: Open modules & open packages

[mark.reinhold at oracle.com]

Issue summary
-------------

  #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
  libraries require reflective access to members of the non-exported
  types of other modules; examples include dependency injection (Guice),
  persistence (JPA), debugging tools, code-automation tools, and
  serialization (XStream).  In some cases the particular library to be
  used is not known until run time (e.g., Hibernate and EclipseLink both
  implement JPA).  This capability is also sometimes used to work around
  bugs in unchangeable code.  Access to non-exported packages can, at
  present, only be done via command-line flags, which is extremely
  awkward.  Provide an easier way for reflective code to access such
  non-exported types. [1]

  #AwkwardStrongEncapsulation --- A non-public element of an exported
  package can still be accessed via the `AccessibleObject::setAccessible`
  method of the core reflection API.  The only way to strongly
  encapsulate such an element is to move it to a non-exported package.
  This makes it awkward, at best, to encapsulate the internals of a
  package that defines a public API. [2]

Proposal
--------

(This is the third proposal for #ReflectiveAccessToNonExportedTypes.
 The note that follows [3] summarizes the background and relates the
 alternatives presented thus far.)

Extend the language of module declarations with the concept of _open_
modules.  Open modules make it easy to define modules whose internals
will be accessed at run time by reflection-based frameworks.  Every
package in an open module has the following properties, by default:

  (1) At compile time it is not exported.

  (2) At run time it is exported without regard to split-package
      conflicts or other inconsistencies, i.e., it does not affect
      the operation of the resolver, nor the construction of layer
      configurations, but is simply exported as if by invoking the
      `Module::addExports` method during layer instantiation.

  (3) At run time it is available for _deep reflection_, i.e., all
      of its elements are accessible via the core reflection API
      including non-public elements, which can be accessed via the
      `AccessibleObject::setAccessible` method.

A package in an open module can be exported explicitly, for use at
compile time, via the familiar `exports` directive, so property (1) will
not hold.  In that case it will also partake in run-time resolution, so
property (2) will not hold either.

Suppose we have a module `foo.bar` that contains an API package
`com.foo.bar`, whose public types are intended for use by other modules,
and a non-API package `com.foo.bar.model`, that contains entity classes
to be manipulated by Hibernate via core reflection.  Then the module
declaration

    open module foo.bar {
        exports com.foo.bar;
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

makes all elements, public and otherwise, of all packages available for
deep reflection at run time, but makes only the public and protected
types in `com.foo.bar` accessible at compile time.  Thus Hibernate can
access non-public elements of the `com.foo.bar.model` package via the
`setAccessible` method, and some other module that `requires foo.bar`
can be compiled against the API in the `com.foo.bar` package but not
against the types in any other package.

Open modules can be considered an intermediate step on the migration
path, between automatic modules and normal modules:

  - An automatic module offers the traditional level of encapsulation:
    All packages are both open for deep reflective access and exported
    for ordinary compile-time and run-time access to their public types.

  - An open module offers a modest degree of stronger encapsulation: All
    packages are open for deep reflective access, but the module's author
    must specify which packages, if any, are exported for ordinary
    compile-time and run-time access.

  - A normal module offers the strongest encapsulation: The module's
    author must specify which packages, if any, are open, or exported,
    or both.

An open module is a good starting point for application code that's being
modularized by its author.  It affords a separation of concerns, so that
the author can focus on defining the module's domain-specific API without
having to worry about how the module will be inspected or manipulated at
run time by reflective frameworks.

                                  * * *

To open specific packages in normal module declarations we introduce a
new per-package directive, `opens`.  A package in a normal module can be
opened, or exported, or both, as follows:

  - If a package is only opened (`opens p;`) then properties (1), (2),
    and (3) hold, i.e., its types are not accessible at compile time,
    it does not affect resolution, and all of its elements are available
    for deep reflection at run time.

  - If a package is only exported (`exports p;`) then those properties
    do not hold, and it is exported exactly as it is today.

  - If a package is both exported and opened (`exports p; opens p;`)
    then it is exported as today and is, additionally, available for
    deep reflection at run time (3).

We can use the `opens` directive to refine the previous example so that
only the `com.foo.bar` package is accessible at compile time, only the
`com.foo.bar.model` package is available for deep reflection at run time,
and the non-public elements of all other packages are strongly
encapsulated:

    module foo.bar {
        exports com.foo.bar;
        opens com.foo.bar.model;
        requires hibernate.core;
        requires hibernate.entitymanager;
    }

It's possible to both open and export a package in a normal module, but
it's often inadvisable.  This is especially so for API packages, since
normally an API's internal implementation details should be strongly
encapsulated.  This combination of directives may, however, be useful for
legacy APIs whose internals are known to be accessed by existing code.

To ensure the integrity of the platform we expect that all the modules of
the JDK itself will be normal modules and that very few, if any, packages
will be opened.

The `opens` directive cannot be used in an `open` module, since all
packages in such modules are implicitly open.

                                  * * *

Both the `exports` and `opens` directives can be qualified, so that a
package is exported or opened only to certain other named modules.  As
before, duplicate directives are not permitted in order to ensure easy
readability.  At most one `exports` directive is relevant to any given
package, and at most one `opens` directive is relevant to any given
package.

There is no syntax for wildcards.  If the defaults are not sufficient for
a package then the package must be named explicitly.

                                  * * *

The existing syntax of `requires public` has long been confusing, so we
here take the opportunity to fix that problem by renaming the `public`
modifier in `requires` directives to `transitive`.  Thus the declaration

    module foo.bar {
        exports com.foo.bar;
        requires public java.sql;
    }

is now written

    module foo.bar {
        exports com.foo.bar;
        requires transitive java.sql;
    }

This is potentially confusing in a different way, since in mathematics
the term "transitive" is usually applied to an entire relation rather
than to three specific elements of a set.  Its use here does not, in
particular, mean that the resolver does not interpret plain `requires`
directives when computing the transitive closure of a set of root
modules.  "Transitive" as used here is in the more abstract sense,
expressing the notion of conveying a property -- in this case, the
readability of the required module -- from one thing to another.  On
balance, however, `transitive` does appear superior to `public`.

Notes
-----

  - This proposal is significantly different from both the first and
    second proposals [4][5].  It recognizes that, in practice, it will
    be common to have open modules, all of whose content is available
    for deep reflection but otherwise inaccessible at compile time unless
    explicitly exported.  If finer-grained control over whether a package
    is opened, exported, or both is required then a normal module
    declaration can be used.

  - This proposal is similar to one made by Stephen Colebourne [6],
    in which he uses the term `exposes`, and to various unpublished
    proposals (of which there have been many!).  We here adopt the term
    `opens` rather than `exposes` in order to avoid confusion between
    the phonetically-similar `exports` and `exposes`, especially for
    non-native speakers of English.  We also use `open` to modify the
    `module` keyword rather than introduce wildcards for the `opens`
    directive, since the former is easier to explain.  The latter would,
    further, lead people to expect wildcards for the `exports` directive,
    but we'd prefer to avoid that since it could encourage the careless
    exportation of all of a module's packages.  Exporting a package for
    use as an API should always be an explicit, affirmative choice.

  - A separate `opens` directive, and the avoidance of wildcards,
    simplifies the interactions amongst qualified and unqualified
    package-level directives.  We no longer need complex rules such as
    those shown for the `private` modifier in the previous proposal [5].

  - This proposal implies corresponding changes to the `ModuleDescriptor`
    and `Module` APIs, but these are reasonably straightforward.

  - This proposal amends the proposal for #ResourceEncapsulation [6]
    to replace the use of private exports with open modules and open
    packages.  Wherever that proposal indicates that a resource can be
    located when its effective package name identifies a package that's
    exported privately, assume instead that the resource can be located
    when that package is either an element of an open module or an open
    package in a normal module.

  - If a container is to ensure that a package in an application module
    is available for deep reflection only by a trusted framework module
    then it can arrange for that by rewriting that module's descriptor,
    as suggested previously [8], to insert the appropriate qualified
    `opens` directive.  This works even in the case, pointed out by Jason
    Greene [9], of two modules that open a package of the same name to
    the same framework module, since opened and unexported packages are
    not subject to the usual package-consistency checks.  There is no
    need any longer for the resolution algorithm to take this type of
    scenario into account [a].  A more flexible way to arrange for such
    precisely-constrained access is a separate issue and may be addressed
    by a future proposal.

  - This proposal primarily addresses "friendly" uses of reflection, such
    as dependency injection and persistence, in which the author of a
    module knows in advance that one or more, or possibly all, packages
    must be opened at run time for deep reflective access by frameworks.
    Intrusive access to arbitrary packages of arbitrary modules by, e.g.,
    serialization frameworks or debugging tools, will still require the
    use of sharp knives such as `--add-exports` or `--add-opens`
    command-line options, the legacy unsupported `sun.misc.Unsafe` API
    and related APIs, or JVM TI.

  - Using `--add-exports` or `--add-opens` command-line options, or their
    equivalents, remains awkward, but sometimes they're the only way out.
    To ease migration it's worth considering some way for an application
    packaged as a JAR file to include such options in its `MANIFEST.MF`
    file, as suggested by Simon Nash [b].  This is addressed in the
    proposal for #AddExportsInManifest [c], which is hereby amended to
    rename the `Add-Exports-Private` attribute to `Add-Opens`.


[1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
[2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
[3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html
[4] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
[5] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000390.html
[6] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000392.html
[7] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html
[8] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
[9] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
[a] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
[b] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
[c] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000391.html