Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports
David M. Lloyd
david.lloyd at redhat.com
Fri Sep 16 15:30:41 UTC 2016
After a fairly detailed review of this proposal, we have determined that
it is not acceptable to Red Hat in its present form. I will list the
primary problems here, and then I'll start up discussion on jigsaw-dev
of several possible solutions that could work for us. I'll number these
in case anyone wants to respond piece-wise.
#1) The "weak" designation appears to be pejorative
Under this solution, many existing frameworks and/or the modules which
consume them must be marked "weak" in order to work correctly, or else a
much more complex module descriptor must be used. While this seems to
be a stepping stone for migration, we find that it rather makes
middleware appear as a second-class citizen.
#2) Weak modules are, in fact, weak
The idea of retaining the Java 8 rules for transition modules appears
good on the surface, however the problem is that users may be forced to
use weak modules indefinitely in order to interoperate with certain
existing middleware and libraries, or else be faced with a potentially
complex migration. If the security benefits of "strong" modules are
real, then users will be rightly inclined to move from weak modules to
strong modules. This can be severely problematic if existing
middleware, libraries, and large applications cannot easily operate as
or with "strong" modules or cannot do so in an "easy to learn"/"easy to
use" fashion. This also means that "weak" modules exist only for the
purposes of maintaining a transition period - at some point they will
inevitably transition from "convenience" to "attractive nuisance". We
believe that it is feasible to solve compatibility to a satisfactory
degree without relegating users of existing software to a lower tier of
security or functionality.
#3) Controlling reflection access is always bound up with controlling
exports
The proposed exports system is simplistic but not exactly simple, yet
several useful modes remain left unaccounted-for. Internally we
examined the usefulness of controlling access to public and so-called
"deep" reflection in combination with whether or not a member was
exported, and we came up with this logical table:
+------------------+---------------------------+
| | Module is... |
| Reflection is... +------------+--------------+
| | Exported | Not Exported |
+----------------------------------------------+
| Forbidden | Not Useful | Useful |
+----------------------------------------------+
| Public Only | Useful | Useful |
+----------------------------------------------+
| Public & Private | Useful | Useful |
+------------------+------------+--------------+
Some of these useful modes seem not to be available, or are not usable
in an obvious or simple manner. These modes are useful, possibly even
necessary, to introduce in Java 9 in order to provide a good
compatibility story with the many existing frameworks in the wild, in
the absence of weak modules.
It has been observed that in some cases, the necessity of granting
public or private reflective access to a module is actually bound not
with what that module exports, but rather with what it requires or uses,
which might be useful in consideration of a solution; for example,
"using" a persistence framework which operates only on public types
would imply that the module would grant back public-only reflective
access to the framework which provides the service.
#4) There isn't always a container
The justification for not providing an easy mechanism to restrict
private reflective access to a trusted framework is that a container can
always rewrite a module or inject classes. However there isn't always a
container present - something which the existence of weak modules
acknowledges. Some frameworks and middleware are meant to be usable
with or without a container, and it is undesirable to force these use
cases into a weak module situation or to require a strong coupling in
the module descriptor in this case. Some other solution should be found
that works both for containers and "regular" applications.
Broadly speaking, a container would be able to do additional things that
are useful do to specifically in a container. But a container should
not be _necessary_ in order to do things that are generally useful, at
least to the greatest extent possible.
#6) Not all reflection is 100% "friendly", and that's OK
Serialization frameworks generally need private reflective access to the
module which contains the classes being serialized. For some
frameworks, this access will already be adequate. Some existing
frameworks utilize Unsafe to create uninitialized instances. However
other frameworks, including frameworks that seek to comply with the
Serialization Specification [1], use ReflectionFactory, as that class
provides (exclusively, as far as I know) the ability to acquire a
constructor that produces a _partially_-initialized class where only the
constructors of superclasses to a certain depth are called. It is our
view that it would be a regression to force such frameworks to use
--add-exports or JVMTI, and using Unsafe for this purpose, or continuing
to use it, (as I understand it) puts at risk the ability to perform
optimizations which depend on entry point control, which I am given to
understand are a strong motivator for many of the design constraints
that exist on the Jigsaw implementation. It should be possible for
these frameworks to continue to use ReflectionFactory (unsupported
though it may be) if suitably privileged (in the security manager
sense), as long as all of the module(s) upon which the framework
reflects have granted private reflective access to it (in the module
declaration sense). This may be imperfect in the event that class
hierarchies are spread across several modules, but that is something
that can be hashed out in detail on jigsaw-dev.
A better alternative to ReflectionFactory itself could be sought in
future releases, if necessary, though one hopes that traditional
serialization will die out in favor of more modern approaches, rendering
the issue less relevant over time. Admittedly, maybe a very long period
of time.
#7) More loose coupling seems necessary and useful
In order for typical applications to function with modern middleware as
modules, without compromising security, it may be necessary to enhance
the loose coupling mechanism (uses/provides), or to provide an
additional, similar mechanism, which allows a symbolic coupling which
would allow modules to declare (in an abstract manner) modules which
need to have reflective access to it in a "friendly" manner.
Implementation ideas are forthcoming on jigsaw-dev.
Providing a way within the system to grant public and/or private
reflection access in a specific manner, and doing so in a manner which
is easy to use and not prejudicial against existing or future middleware
and small or large applications which consume such middleware in the
Java SE or EE space, is our most basic requirement for satisfactory
resolution of this issue. I will follow up on jigsaw-dev with some
implementation thoughts and more specific requirements.
This list may not be exhaustive but it should be, at the least, a very
good starting point for discussion, hopefully with a short path to
implementation.
On 09/12/2016 10:08 AM, Mark Reinhold wrote:
> Issue summary
> -------------
>
> #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
> libraries require reflective access to members of the non-exported
> types of other modules; examples include dependency injection (Guice),
> persistence (JPA), debugging tools, code-automation tools, and
> serialization (XStream). In some cases the particular library to be
> used is not known until run time (e.g., Hibernate and EclipseLink both
> implement JPA). This capability is also sometimes used to work around
> bugs in unchangeable code. Access to non-exported packages can, at
> present, only be done via command-line flags, which is extremely
> awkward. Provide an easier way for reflective code to access such
> non-exported types. [1]
>
> #AwkwardStrongEncapsulation --- A non-public element of an exported
> package can still be accessed via the `AccessibleObject::setAccessible`
> method of the core reflection API. The only way to strongly
> encapsulate such an element is to move it to a non-exported package.
> This makes it awkward, at best, to encapsulate the internals of a
> package that defines a public API. [2]
>
> Proposal
> --------
>
> (Warning: This is somewhat long, and in the end it affects both `exports`
> and `requires` directives.)
>
> Extend the language of module declarations with the concept of _weak
> modules_. Weak modules make it easy to modularize components whose
> internals will be accessed by reflection-based frameworks. Every
> package in a weak module has the following properties:
>
> (A) It is exported at both compile time and run time, as if by an
> `exports` directive, and
>
> (B) Its non-public elements are available for _deep_ reflection, i.e.,
> at run time they can be made accessible to code outside the module
> via the `AccessibleObject::setAccessible` method of the core
> reflection API.
>
> In other words, every type defined in a weak module, whether public or
> not, is subject to exactly the same access checks as in Java SE 8 and
> earlier releases.
>
> A weak module is declared by placing the modifier `weak` before the
> `module` keyword. The declaration of a weak module cannot contain any
> explicit `exports` directives. If the `weak` modifier does not appear
> before the `module` keyword then the declared module is _strong_, and
> it can contain explicit `exports` directives.
>
> Suppose we have a module `foo.bar` which has an internal package
> `com.foo.bar.model` that contains entity classes to be manipulated by
> Hibernate, via core reflection. Then the module declaration
>
> weak module foo.bar {
> // No exports
> requires hibernate.core;
> requires hibernate.entitymanager;
> }
>
> exports the public types in `com.foo.bar.model`, and those of any other
> packages, in all phases. It additionally makes all non-public elements
> of all packages available for deep reflection, enabling Hibernate to
> access such elements in the `com.foo.bar.model` package via the
> `setAccessible` method.
>
> Weak modules simplify the process of migrating to modules. The steps
> to convert an existing component into a module were, previously:
>
> (1) Make any changes necessary to get it working as an automatic
> module (e.g., eliminate duplicate packages), and then
>
> (2) Write an explicit module declaration, which entails identifying
> both the component's dependences (`requires`) and the packages
> whose public types are to be made available to other modules
> (`exports`).
>
> With weak modules we can now divide the second step into two steps:
>
> (2a) Write an explicit module declaration for a weak module, which
> entails identifying just the component's dependences (`requires`).
>
> (2b) Convert the weak module into a strong module, which entails
> identifying the packages of the component whose public types
> are to be made available to other modules (`exports`).
>
> In other words, weak modules make it possible to focus first upon the
> reliable configuration of a module (`requires`), and then later think
> about its strong encapsulation (`exports`).
>
> Weak modules are "weak" in what they export, but they remain subject
> to all of the constraints required to achieve reliable configuration.
> They do not read the unnamed module (i.e., the class path), they do not
> allow cycles in the module graph, and they do not allow split packages.
> Weak modules read named modules only as indicated by their `requires`
> directives, and they consume and provide services only as indicated by
> their `uses` and `provides` directives.
>
> * * *
>
> In a strong module, an ordinary `exports` directive exports a package at
> both compile time and run time (property (A) above) but does not make its
> non-public types available for deep reflection (B). In order to enable a
> package in a strong module to be exported in the same way as in a weak
> module we introduce the per-export modifier `private` to denote this
> second property.
>
> If the above weak `foo.bar` module, e.g., contains some other packages
> besides `com.foo.bar.model`, and we wish to encapsulate those packages,
> we can convert it into a strong module with the declaration
>
> module foo.bar {
> exports private com.foo.bar.model;
> requires hibernate.core;
> requires hibernate.entitymanager;
> }
>
> Now Hibernate can still access any public or non-public entity classes in
> the `com.foo.bar.model` package, but all the other packages are strongly
> encapsulated.
>
> The `private` modifier should generally not be used to export a package
> containing an API, since normally an API's internal implementation
> details should be strongly encapsulated. It may, however, be useful for
> legacy APIs whose internals are known to be accessed by existing code.
>
> Every package in a weak module, an automatic module, or an unnamed module
> is exported as if by an `exports private` directive.
>
> To ensure the integrity of the platform we expect few, if any, packages
> in the JDK itself to be exported with the `private` modifier.
>
> * * *
>
> The new `private` modifier can also be used with qualified exports,
> though they interact with unqualified exports in a non-obvious way.
>
> - If you write `exports p` then you can also write `exports private
> p to m`, so that code in module `m` can access the non-public types
> of `p` via deep reflection but code outside of `m` can only access
> the public types of `p`.
>
> - If you write `exports p to m1` then you can also write `exports
> private p to m2`, so that code in `m2` can access the non-public
> types of `p` via deep reflection, code in `m1` can access the
> public types of `p`, but no code in any other module can access
> any of the types of `p`.
>
> - If you write `exports private p` then you cannot also have a
> qualified export of `p`, since code in all other modules already
> has access to the non-public types of `p` via deep reflection.
>
> Put informally, you can give your friends additional access, but you
> can't discriminate against them by giving them less access than everyone
> else.
>
> As before, duplicate `exports` directives are not permitted, in order to
> ensure easy readability. At most one `exports` directive is relevant to
> any given package/module pair, and it's easy to determine which one.
>
> * * *
>
> The introduction of `private` as a modifier of `exports` directives calls
> the existing syntax of `requires public` even more strongly into question
> than before. A module declaration of the form
>
> module foo.bar {
> exports private com.foo.bar.baz;
> requires public java.sql;
> }
>
> is likely to be very confusing to an uninformed reader. The `private`
> modifier in the `exports` directive means that the private elements of
> the `com.foo.bar.baz` package are exported for deep reflection at run
> time. The `public` modifier in the `requires` directive, however, does
> not mean that the public elements of the `java.sql` module are needed by
> this module; that is true of any plain `requires` directive. It means
> that, additionally, any client of this module is granted implied
> readability to the `java.sql` module, thereby gaining access to all of
> its exported types.
>
> To reduce this confusion we rename the `public` modifier in `requires`
> directives to `transitive`. Thus the above example becomes
>
> module foo.bar {
> exports private com.foo.bar.baz;
> requires transitive java.sql;
> }
>
> This is potentially confusing in a different way, since in mathematics
> the term "transitive" is usually applied to an entire relation rather
> than to three specific elements of a set. Its use here does not, in
> particular, mean that the resolver does not interpret plain `requires`
> directives when computing the transitive closure of a set of root
> modules. "Transitive" as used here is in the more abstract sense,
> expressing the notion of conveying a property -- in this case, the
> readability of the required module -- from one thing to another.
>
>
> Notes
> -----
>
> - This is significantly different from the first proposal [3]. It adds
> the notion of weak modules, to ease migration, and also the notion of
> exporting a package without enabling deep reflection, to strengthen
> encapsulation.
>
> - This proposal removes the notion of dynamic exports, which in the
> presence of private exports would introduce considerable complexity
> into the interactions between qualified and unqualified exports.
> This means that it is no longer possible to export a package only at
> run time, so it is no longer possible for the author of a module to
> express the intent that the types of a non-API package are meant to
> be available to frameworks for deep reflection at run time but
> inaccessible at compile time. The dynamic-export feature could, if
> needed, be added in a future release.
>
> - A strong module with no exports makes no types accessible to code in
> other modules while a weak module makes all of its types accessible,
> both directly and via deep reflection. The declarations of such
> modules are, however, visually similar since most of their text lies
> between the curly braces:
>
> module m1 {
> requires ...;
> uses ...;
> provides ...;
> }
>
> weak module m2 {
> requires ...;
> uses ...;
> provides ...;
> }
>
> We suspect that this visual similarity will not cause much confusion
> in practice since strong modules that export no packages will be very
> rare.
>
> - If a container is to ensure that a package in an application module
> is available for deep reflection only by a trusted framework then it
> can arrange for that by rewriting that module's descriptor, as
> suggested previously [4], to insert the appropriate qualified private
> export. If there is a possibility that two modules will need to
> export a package of the same name to the same framework module, as
> suggested by Jason Greene [5], then the container should instead
> inject a small class into each module whose static initializer
> invokes the `Module::addExports` method in order to export the
> package to the framework module. There is no need any longer for
> the resolution algorithm to take this scenario into account [6].
>
> - This proposal primarily addresses "friendly" uses of reflection, such
> as dependency injection and persistence, in which the author of a
> module knows in advance that one or more packages must be exported at
> run time for deep reflective access by frameworks. Intrusive access
> to arbitrary packages of arbitrary modules by, e.g., serialization
> frameworks or debugging tools, will still require the use of sharp
> knives such as the `--add-exports` command-line option, the legacy
> unsupported `sun.misc.Unsafe` API and related APIs, or JVM TI.
>
> - Using the `--add-exports` option or its equivalent remains awkward,
> and sometimes it's the only way out. To ease migration it's worth
> considering some way for an application packaged as a JAR file to
> include such options in its `MANIFEST.MF` file, as suggested by Simon
> Nash [7]. This is tracked as #AddExportsInManifest [8].
>
>
> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
> [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
> [3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
> [4] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
> [5] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
> [6] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
> [7] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
> [8] http://openjdk.java.net/projects/jigsaw/spec/issues/#AddExportsInManifest
>
--
- DML
More information about the jpms-spec-experts
mailing list