Alternatives and fixes to #ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation proposals
David M. Lloyd
david.lloyd at redhat.com
Fri Sep 16 18:44:22 UTC 2016
I've recently sent the below message to the jpms-spec-experts list. For
the reasons listed below, we find the current proposals unworkable, and
would like to propose some refinements and/or alternatives. I'll follow
up soon with separate emails to discuss specific alternatives and fixes
in various areas, and to cover what our minimum requirements are in each
area.
On 09/16/2016 10:30 AM, David M. Lloyd wrote:
> After a fairly detailed review of this proposal, we have determined that
> it is not acceptable to Red Hat in its present form. I will list the
> primary problems here, and then I'll start up discussion on jigsaw-dev
> of several possible solutions that could work for us. I'll number these
> in case anyone wants to respond piece-wise.
>
> #1) The "weak" designation appears to be pejorative
>
> Under this solution, many existing frameworks and/or the modules which
> consume them must be marked "weak" in order to work correctly, or else a
> much more complex module descriptor must be used. While this seems to
> be a stepping stone for migration, we find that it rather makes
> middleware appear as a second-class citizen.
>
> #2) Weak modules are, in fact, weak
>
> The idea of retaining the Java 8 rules for transition modules appears
> good on the surface, however the problem is that users may be forced to
> use weak modules indefinitely in order to interoperate with certain
> existing middleware and libraries, or else be faced with a potentially
> complex migration. If the security benefits of "strong" modules are
> real, then users will be rightly inclined to move from weak modules to
> strong modules. This can be severely problematic if existing
> middleware, libraries, and large applications cannot easily operate as
> or with "strong" modules or cannot do so in an "easy to learn"/"easy to
> use" fashion. This also means that "weak" modules exist only for the
> purposes of maintaining a transition period - at some point they will
> inevitably transition from "convenience" to "attractive nuisance". We
> believe that it is feasible to solve compatibility to a satisfactory
> degree without relegating users of existing software to a lower tier of
> security or functionality.
>
> #3) Controlling reflection access is always bound up with controlling
> exports
>
> The proposed exports system is simplistic but not exactly simple, yet
> several useful modes remain left unaccounted-for. Internally we
> examined the usefulness of controlling access to public and so-called
> "deep" reflection in combination with whether or not a member was
> exported, and we came up with this logical table:
>
> +------------------+---------------------------+
> | | Module is... |
> | Reflection is... +------------+--------------+
> | | Exported | Not Exported |
> +----------------------------------------------+
> | Forbidden | Not Useful | Useful |
> +----------------------------------------------+
> | Public Only | Useful | Useful |
> +----------------------------------------------+
> | Public & Private | Useful | Useful |
> +------------------+------------+--------------+
>
> Some of these useful modes seem not to be available, or are not usable
> in an obvious or simple manner. These modes are useful, possibly even
> necessary, to introduce in Java 9 in order to provide a good
> compatibility story with the many existing frameworks in the wild, in
> the absence of weak modules.
>
> It has been observed that in some cases, the necessity of granting
> public or private reflective access to a module is actually bound not
> with what that module exports, but rather with what it requires or uses,
> which might be useful in consideration of a solution; for example,
> "using" a persistence framework which operates only on public types
> would imply that the module would grant back public-only reflective
> access to the framework which provides the service.
>
> #4) There isn't always a container
>
> The justification for not providing an easy mechanism to restrict
> private reflective access to a trusted framework is that a container can
> always rewrite a module or inject classes. However there isn't always a
> container present - something which the existence of weak modules
> acknowledges. Some frameworks and middleware are meant to be usable
> with or without a container, and it is undesirable to force these use
> cases into a weak module situation or to require a strong coupling in
> the module descriptor in this case. Some other solution should be found
> that works both for containers and "regular" applications.
>
> Broadly speaking, a container would be able to do additional things that
> are useful do to specifically in a container. But a container should
> not be _necessary_ in order to do things that are generally useful, at
> least to the greatest extent possible.
>
> #6) Not all reflection is 100% "friendly", and that's OK
>
> Serialization frameworks generally need private reflective access to the
> module which contains the classes being serialized. For some
> frameworks, this access will already be adequate. Some existing
> frameworks utilize Unsafe to create uninitialized instances. However
> other frameworks, including frameworks that seek to comply with the
> Serialization Specification [1], use ReflectionFactory, as that class
> provides (exclusively, as far as I know) the ability to acquire a
> constructor that produces a _partially_-initialized class where only the
> constructors of superclasses to a certain depth are called. It is our
> view that it would be a regression to force such frameworks to use
> --add-exports or JVMTI, and using Unsafe for this purpose, or continuing
> to use it, (as I understand it) puts at risk the ability to perform
> optimizations which depend on entry point control, which I am given to
> understand are a strong motivator for many of the design constraints
> that exist on the Jigsaw implementation. It should be possible for
> these frameworks to continue to use ReflectionFactory (unsupported
> though it may be) if suitably privileged (in the security manager
> sense), as long as all of the module(s) upon which the framework
> reflects have granted private reflective access to it (in the module
> declaration sense). This may be imperfect in the event that class
> hierarchies are spread across several modules, but that is something
> that can be hashed out in detail on jigsaw-dev.
>
> A better alternative to ReflectionFactory itself could be sought in
> future releases, if necessary, though one hopes that traditional
> serialization will die out in favor of more modern approaches, rendering
> the issue less relevant over time. Admittedly, maybe a very long period
> of time.
>
> #7) More loose coupling seems necessary and useful
>
> In order for typical applications to function with modern middleware as
> modules, without compromising security, it may be necessary to enhance
> the loose coupling mechanism (uses/provides), or to provide an
> additional, similar mechanism, which allows a symbolic coupling which
> would allow modules to declare (in an abstract manner) modules which
> need to have reflective access to it in a "friendly" manner.
> Implementation ideas are forthcoming on jigsaw-dev.
>
> Providing a way within the system to grant public and/or private
> reflection access in a specific manner, and doing so in a manner which
> is easy to use and not prejudicial against existing or future middleware
> and small or large applications which consume such middleware in the
> Java SE or EE space, is our most basic requirement for satisfactory
> resolution of this issue. I will follow up on jigsaw-dev with some
> implementation thoughts and more specific requirements.
>
> This list may not be exhaustive but it should be, at the least, a very
> good starting point for discussion, hopefully with a short path to
> implementation.
>
> On 09/12/2016 10:08 AM, Mark Reinhold wrote:
>> Issue summary
>> -------------
>>
>> #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
>> libraries require reflective access to members of the non-exported
>> types of other modules; examples include dependency injection (Guice),
>> persistence (JPA), debugging tools, code-automation tools, and
>> serialization (XStream). In some cases the particular library to be
>> used is not known until run time (e.g., Hibernate and EclipseLink both
>> implement JPA). This capability is also sometimes used to work around
>> bugs in unchangeable code. Access to non-exported packages can, at
>> present, only be done via command-line flags, which is extremely
>> awkward. Provide an easier way for reflective code to access such
>> non-exported types. [1]
>>
>> #AwkwardStrongEncapsulation --- A non-public element of an exported
>> package can still be accessed via the `AccessibleObject::setAccessible`
>> method of the core reflection API. The only way to strongly
>> encapsulate such an element is to move it to a non-exported package.
>> This makes it awkward, at best, to encapsulate the internals of a
>> package that defines a public API. [2]
>>
>> Proposal
>> --------
>>
>> (Warning: This is somewhat long, and in the end it affects both `exports`
>> and `requires` directives.)
>>
>> Extend the language of module declarations with the concept of _weak
>> modules_. Weak modules make it easy to modularize components whose
>> internals will be accessed by reflection-based frameworks. Every
>> package in a weak module has the following properties:
>>
>> (A) It is exported at both compile time and run time, as if by an
>> `exports` directive, and
>>
>> (B) Its non-public elements are available for _deep_ reflection, i.e.,
>> at run time they can be made accessible to code outside the module
>> via the `AccessibleObject::setAccessible` method of the core
>> reflection API.
>>
>> In other words, every type defined in a weak module, whether public or
>> not, is subject to exactly the same access checks as in Java SE 8 and
>> earlier releases.
>>
>> A weak module is declared by placing the modifier `weak` before the
>> `module` keyword. The declaration of a weak module cannot contain any
>> explicit `exports` directives. If the `weak` modifier does not appear
>> before the `module` keyword then the declared module is _strong_, and
>> it can contain explicit `exports` directives.
>>
>> Suppose we have a module `foo.bar` which has an internal package
>> `com.foo.bar.model` that contains entity classes to be manipulated by
>> Hibernate, via core reflection. Then the module declaration
>>
>> weak module foo.bar {
>> // No exports
>> requires hibernate.core;
>> requires hibernate.entitymanager;
>> }
>>
>> exports the public types in `com.foo.bar.model`, and those of any other
>> packages, in all phases. It additionally makes all non-public elements
>> of all packages available for deep reflection, enabling Hibernate to
>> access such elements in the `com.foo.bar.model` package via the
>> `setAccessible` method.
>>
>> Weak modules simplify the process of migrating to modules. The steps
>> to convert an existing component into a module were, previously:
>>
>> (1) Make any changes necessary to get it working as an automatic
>> module (e.g., eliminate duplicate packages), and then
>>
>> (2) Write an explicit module declaration, which entails identifying
>> both the component's dependences (`requires`) and the packages
>> whose public types are to be made available to other modules
>> (`exports`).
>>
>> With weak modules we can now divide the second step into two steps:
>>
>> (2a) Write an explicit module declaration for a weak module, which
>> entails identifying just the component's dependences (`requires`).
>>
>> (2b) Convert the weak module into a strong module, which entails
>> identifying the packages of the component whose public types
>> are to be made available to other modules (`exports`).
>>
>> In other words, weak modules make it possible to focus first upon the
>> reliable configuration of a module (`requires`), and then later think
>> about its strong encapsulation (`exports`).
>>
>> Weak modules are "weak" in what they export, but they remain subject
>> to all of the constraints required to achieve reliable configuration.
>> They do not read the unnamed module (i.e., the class path), they do not
>> allow cycles in the module graph, and they do not allow split packages.
>> Weak modules read named modules only as indicated by their `requires`
>> directives, and they consume and provide services only as indicated by
>> their `uses` and `provides` directives.
>>
>> * * *
>>
>> In a strong module, an ordinary `exports` directive exports a package at
>> both compile time and run time (property (A) above) but does not make its
>> non-public types available for deep reflection (B). In order to enable a
>> package in a strong module to be exported in the same way as in a weak
>> module we introduce the per-export modifier `private` to denote this
>> second property.
>>
>> If the above weak `foo.bar` module, e.g., contains some other packages
>> besides `com.foo.bar.model`, and we wish to encapsulate those packages,
>> we can convert it into a strong module with the declaration
>>
>> module foo.bar {
>> exports private com.foo.bar.model;
>> requires hibernate.core;
>> requires hibernate.entitymanager;
>> }
>>
>> Now Hibernate can still access any public or non-public entity classes in
>> the `com.foo.bar.model` package, but all the other packages are strongly
>> encapsulated.
>>
>> The `private` modifier should generally not be used to export a package
>> containing an API, since normally an API's internal implementation
>> details should be strongly encapsulated. It may, however, be useful for
>> legacy APIs whose internals are known to be accessed by existing code.
>>
>> Every package in a weak module, an automatic module, or an unnamed module
>> is exported as if by an `exports private` directive.
>>
>> To ensure the integrity of the platform we expect few, if any, packages
>> in the JDK itself to be exported with the `private` modifier.
>>
>> * * *
>>
>> The new `private` modifier can also be used with qualified exports,
>> though they interact with unqualified exports in a non-obvious way.
>>
>> - If you write `exports p` then you can also write `exports private
>> p to m`, so that code in module `m` can access the non-public types
>> of `p` via deep reflection but code outside of `m` can only access
>> the public types of `p`.
>>
>> - If you write `exports p to m1` then you can also write `exports
>> private p to m2`, so that code in `m2` can access the non-public
>> types of `p` via deep reflection, code in `m1` can access the
>> public types of `p`, but no code in any other module can access
>> any of the types of `p`.
>>
>> - If you write `exports private p` then you cannot also have a
>> qualified export of `p`, since code in all other modules already
>> has access to the non-public types of `p` via deep reflection.
>>
>> Put informally, you can give your friends additional access, but you
>> can't discriminate against them by giving them less access than everyone
>> else.
>>
>> As before, duplicate `exports` directives are not permitted, in order to
>> ensure easy readability. At most one `exports` directive is relevant to
>> any given package/module pair, and it's easy to determine which one.
>>
>> * * *
>>
>> The introduction of `private` as a modifier of `exports` directives calls
>> the existing syntax of `requires public` even more strongly into question
>> than before. A module declaration of the form
>>
>> module foo.bar {
>> exports private com.foo.bar.baz;
>> requires public java.sql;
>> }
>>
>> is likely to be very confusing to an uninformed reader. The `private`
>> modifier in the `exports` directive means that the private elements of
>> the `com.foo.bar.baz` package are exported for deep reflection at run
>> time. The `public` modifier in the `requires` directive, however, does
>> not mean that the public elements of the `java.sql` module are needed by
>> this module; that is true of any plain `requires` directive. It means
>> that, additionally, any client of this module is granted implied
>> readability to the `java.sql` module, thereby gaining access to all of
>> its exported types.
>>
>> To reduce this confusion we rename the `public` modifier in `requires`
>> directives to `transitive`. Thus the above example becomes
>>
>> module foo.bar {
>> exports private com.foo.bar.baz;
>> requires transitive java.sql;
>> }
>>
>> This is potentially confusing in a different way, since in mathematics
>> the term "transitive" is usually applied to an entire relation rather
>> than to three specific elements of a set. Its use here does not, in
>> particular, mean that the resolver does not interpret plain `requires`
>> directives when computing the transitive closure of a set of root
>> modules. "Transitive" as used here is in the more abstract sense,
>> expressing the notion of conveying a property -- in this case, the
>> readability of the required module -- from one thing to another.
>>
>>
>> Notes
>> -----
>>
>> - This is significantly different from the first proposal [3]. It adds
>> the notion of weak modules, to ease migration, and also the notion of
>> exporting a package without enabling deep reflection, to strengthen
>> encapsulation.
>>
>> - This proposal removes the notion of dynamic exports, which in the
>> presence of private exports would introduce considerable complexity
>> into the interactions between qualified and unqualified exports.
>> This means that it is no longer possible to export a package only at
>> run time, so it is no longer possible for the author of a module to
>> express the intent that the types of a non-API package are meant to
>> be available to frameworks for deep reflection at run time but
>> inaccessible at compile time. The dynamic-export feature could, if
>> needed, be added in a future release.
>>
>> - A strong module with no exports makes no types accessible to code in
>> other modules while a weak module makes all of its types accessible,
>> both directly and via deep reflection. The declarations of such
>> modules are, however, visually similar since most of their text lies
>> between the curly braces:
>>
>> module m1 {
>> requires ...;
>> uses ...;
>> provides ...;
>> }
>>
>> weak module m2 {
>> requires ...;
>> uses ...;
>> provides ...;
>> }
>>
>> We suspect that this visual similarity will not cause much confusion
>> in practice since strong modules that export no packages will be very
>> rare.
>>
>> - If a container is to ensure that a package in an application module
>> is available for deep reflection only by a trusted framework then it
>> can arrange for that by rewriting that module's descriptor, as
>> suggested previously [4], to insert the appropriate qualified private
>> export. If there is a possibility that two modules will need to
>> export a package of the same name to the same framework module, as
>> suggested by Jason Greene [5], then the container should instead
>> inject a small class into each module whose static initializer
>> invokes the `Module::addExports` method in order to export the
>> package to the framework module. There is no need any longer for
>> the resolution algorithm to take this scenario into account [6].
>>
>> - This proposal primarily addresses "friendly" uses of reflection, such
>> as dependency injection and persistence, in which the author of a
>> module knows in advance that one or more packages must be exported at
>> run time for deep reflective access by frameworks. Intrusive access
>> to arbitrary packages of arbitrary modules by, e.g., serialization
>> frameworks or debugging tools, will still require the use of sharp
>> knives such as the `--add-exports` command-line option, the legacy
>> unsupported `sun.misc.Unsafe` API and related APIs, or JVM TI.
>>
>> - Using the `--add-exports` option or its equivalent remains awkward,
>> and sometimes it's the only way out. To ease migration it's worth
>> considering some way for an application packaged as a JAR file to
>> include such options in its `MANIFEST.MF` file, as suggested by Simon
>> Nash [7]. This is tracked as #AddExportsInManifest [8].
>>
>>
>> [1]
>> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes
>>
>> [2]
>> http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation
>>
>> [3]
>> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html
>>
>> [4]
>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html
>> [5]
>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html
>> [6]
>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html
>> [7]
>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html
>>
>> [8]
>> http://openjdk.java.net/projects/jigsaw/spec/issues/#AddExportsInManifest
>>
>
--
- DML
More information about the jigsaw-dev
mailing list