Proposal: #ReflectiveAccessToNonExportedTypes: `exports dynamic`

Fri Jul 1 09:15:29 UTC 2016

As for JUnit 4 or Unsafe, reflection is used because of an actual need to
interact with these libraries/APIs.

As for JUnit, it is mainly IDE vendors that needed to access internals in
order to display unit testing results and to extract details about testing
results. By making this extraction more awkward, this would not have
changed the fact that IDE-vendors want to interact with the internal APIs.
It  would only have been more difficult to do. In my eyes, limiting
reflection is fighting the symptoms, not the cause. For example, JUnit 5
has adressed this issue by implementing a public API for extracting such
information what immediately solved the current problem. I do not think
anybody prefers reflection over an official API if the latter one satisfies
all needs.

The same goes for Unsafe. I do not think anybody likes to use Unsafe if
there is a better alternative. The Unsafe API is awkward and - obviously -
unsafe. And despite the implementors attempt to shield the API to bootstrap
classes, people found their ways around because there has not been another
way to make certain things work. I am sure, the more the JVM offers
adequate alternatives, people will migrate happily.

Also, as a thought experiment, let us say accessing internal APIs had been
impossible in the past: we would have missed out on Hadoop, Cassandra,
Mockito, Spark; and only in 2016 we could start running JUnit from our IDE.
I think we can all agree that this is not the outcome we would have wanted.
Reflection - despite it breaking encapsulation - has done more good then
bad.

As for the argument that required dynamic exports should be documented;
ideally yes. Equally, APIs should be documented but many librararies fail
to offer good documentation, especially in the open-source space where time
is sparse. I already see myself answering questions to puzzled junior
developers that know Java for a few months that "this is required because
this library uses reflection" where they do not even know what the latter
is. Effectively, building this barrier makes Java more difficult to use and
I am convinced that many people will adapt adding dynamic exports for any
package as some form of "best practice" to get on with their lifes. I will
also bet that someone will deliver some "auto-export Maven plugin" to
automate this "best practive".

Finally, I do not believe the service loader API offers an adequate
alternative to many use-cases of reflection (while it can perfectly
substitute some of them). Many state-of-the art libraries actually
intentionally offer POJO approaches without requiring their users to create
interfaces for any shared interaction. I do not think that a gentle push
into a direction that contradicts most existing Java libraries would be a
taken well by the people implementing this software and in the end, the
module system will be as successful as its adoption. I rather see people
finding new ways around restrictions to keep things simple (for themselves).

For all these reasons I do not think that it is a good idea to limit
reflection to exported packages.

2016-07-01 9:27 GMT+02:00 Nicolai Parlog <nipa at codefx.org>:

>  Hi!
>
> > I do not think that shielding internals is a problem with regards
> > to reflection.
>
> I think it is. I've not been emerged in the community for very long so
> I know only two examples (but still): Unsafe and JUnit 4. In both
> cases internals were accessed via reflection. And in both cases
> dependencies became so ubiquitous that changing any details became
> impossible for the maintainers.
>
> I think the outcry when Unsafe was planned to go away and the fact
> that JUnit 4 had to be rewritten because evolution was effectively
> prevented by unwanted dependencies make my point here.
>
> > I do neither think that reflection only interests internals.
>
> I agree. Not "only" but "all too often", though.
>
> > For example, maybe you use Hibernate for your persistance layer
> > without exposing those DTOs from your module.
>
> Why would I do that?! This is exactly where reflection got us used to
> sloppy architecture. If a tool uses my DTOs to implement my
> persistence layer, then it looks like those DTOs are part of my API.
> Maybe not public to everyone but qualified and dynamic exports give me
> tools to reduce visibility.
>
> > I do not care about how Hibernate works as long as it does work.
> > Almost any library uses reflection and I would not want to dig
> > into implementation details to find out how to set up my dynamic
> > exports.
>
> Maybe I'm missing something here but to me it looks like "export to a
> module whatever that module is supposed to interact with" is a very
> sensible default that should cover a vast majority of uses cases, no?
>
>         I have DTOs && Hibernate needs DTOs -> export DTOs to Hibernate.
>
> Why would I look at the implementation? And if there were non-obvious
> scenarios they should be properly documented so I can react accordingly.
>
> > As a library implementor, you aim for the biggest convenience.
>
> Maybe we should aim for clarity and well-defined interactions instead.
>
>  so long ... Nicolai
>
>
>
> On 01.07.2016 00:20, Rafael Winterhalter wrote:
> > I disagree on two ends here.
> >
> > I do not think that shielding internals is a problem with regards
> > to reflection. You want to shield internals in order to not break
> > dependants by prohibiting their existance. With reflection, you do
> > however typically react to the actual implementation by discovering
> > it at runtime. If your use pattern was fixed, you would always
> > require a static dependency over reflection in the first place. I
> > do neither think that reflection only interests internals. For
> > example, maybe you use Hibernate for your persistance layer without
> > exposing those DTOs from your module. The problem is that you
> > invert responsibilities. I do not care about how Hibernate works as
> > long as it does work. Almost any library uses reflection and I
> > would not want to dig into implementation details to find out how
> > to set up my dynamic exports. I want to require a module and
> > hopefully the compiler tells me about anything that's wrong.
> > Reflection access errors would only be an awkward side effect that
> > can be avoided by not applying modules alltogether. I think this
> > limitation would therefore hurt Java 9 adoption.
> >
> > The second problem I have is the issue of attempting to fix an old
> > regret. From the perspective of Hibernate, it would be easiest to
> > not attempt modules as their introduction would require its users
> > to apply additional configuration. As a library implementor, you
> > aim for the biggest convenience. Adding this restriction neither
> > adds additional security where a security manager should be used. I
> > again think this would hurt adoption.
> >
> > I agree with Stephen's argument even though I do not think that
> > forbidding dynamic access would be a feature anybody would use so
> > it should not be added. Rather, something should be integrated in
> > the security manager, if this is considered necessary.
> >
> > Cheers, Rafael
> >
> > Am 30.06.2016 11:37 nachm. schrieb "Nicolai Parlog"
> > <nipa at codefx.org>:
> >>
> >> Hi!
> >>
> >>> In my opinion, reflection should not apply any module-boundary
> >>> checks.
> >>
> >> Maybe this list is not a representative selection of Java
> >> developers, maybe I'm just the only one thinking this but, to
> >> paraphrase:
> >>
> >> Reflection over internals must die in a fire!
> >>
> >> Yes, yes, tools to great things, we need to see everything, ...
> >> Still, I think that modules need a safe space that they can use
> >> without anybody intruding. Maybe I'm a hardliner but I already
> >> view the command line parameters with suspicion.
> >>
> >> In that light, I like the "dynamic" proposal.
> >>
> >> Not the name, though. Having the symmetric pair "static vs
> >> dynamic" looks nice but it takes a lot of words to explain why
> >> exactly these terms apply here. We seem to be looking for
> >> qualifiers for "requires" and "exports". Maybe just using other
> >> terms solves the problem?
> >>
> >> requires & consumes exports & contains
> >>
> >> so long ... Nicolai
> >>
> >>
> >>
> >> On 30.06.2016 01:23, Rafael Winterhalter wrote:
> >>> I also agree here that the use of reflection should not require
> >>> any form of explicit permission by the user of a library. In a
> >>> way, this ties an implementation detail (reflection in this
> >>> case) to the usage of the library. The library cannot work
> >>> without the explicit configuration by a user and this only adds
> >>> boilerplate to using Hibernate.
> >>>
> >>> Unfortunately, many Java developers still do not know much
> >>> about reflection or about the inner workings of many libraries
> >>> in the market. If a user needed to explicitly export packages,
> >>> this might just lead to people "over exporting" their packages
> >>> just to "make Hibernate work".
> >>>
> >>> In my opinion, reflection should not apply any module-boundary
> >>> checks.
> >>>
> >>> Best regards, Rafael
> >>>
> >>> 2016-06-29 20:13 GMT+02:00 Paul Benedict
> >>> <pbenedict at apache.org>:
> >>>
> >>>> Replying to "observers" list...
> >>>>
> >>>> I think the main problem here is that core reflection has
> >>>> become too restrictive. Reflection should be "magical" and
> >>>> have no knowledge of any kind of module and/or visibility
> >>>> boundaries. If you really want to control what can be
> >>>> reflected upon, that should be done via the Policy file of
> >>>> the Security Manager.
> >>>>
> >>>> This proposal is trying to hack out-of-the-box created by
> >>>> too strict of a design. It is very awkward to denote which
> >>>> packages must be reflectable ahead of time --- how can you
> >>>> figure that out?? The consumers who are doing the reflection
> >>>> are innumerable. I'd go as far to say it's it's impossible to
> >>>> devise a preemptive comprehensive strategy that may satisfy
> >>>> the kinds of tooling in the wild.
> >>>>
> >>>> It's best just to open up reflection so it can continue
> >>>> doing what it always has done well: let me see anything I
> >>>> want unless the Security Manager says no.
> >>>>
> >>>> Cheers, Paul
> >>>>
> >>>> On Wed, Jun 29, 2016 at 3:51 AM, Remi Forax
> >>>> <forax at univ-mlv.fr> wrote:
> >>>>
> >>>>>
> >>>>>
> >>>>> ----- Mail original -----
> >>>>>> De: "Mark Reinhold" <mark.reinhold at oracle.com> À:
> >>>>>> jpms-spec-experts at openjdk.java.net Envoyé: Mardi 28 Juin
> >>>>>> 2016 23:18:15 Objet: Proposal:
> >>>>>> #ReflectiveAccessToNonExportedTypes: `exports dynamic`
> >>>>>>
> >>>>>> Issue summary -------------
> >>>>>>
> >>>>>> #ReflectiveAccessToNonExportedTypes --- Some kinds of
> >>>>>> framework libraries require reflective access to members
> >>>>>> of the non-exported types of other modules; examples
> >>>>>> include dependency injection
> >>>> (Guice),
> >>>>>> persistence (JPA), debugging tools, code-automation
> >>>>>> tools, and serialization (XStream).  In some cases the
> >>>>>> particular library to be used is not known until run time
> >>>>>> (e.g., Hibernate and EclipseLink
> >>>> both
> >>>>>> implement JPA).  This capability is also sometimes used
> >>>>>> to work
> >>>> around
> >>>>>> bugs in unchangeable code.  Access to non-exported
> >>>>>> packages can, at present, only be done via command-line
> >>>>>> flags, which is extremely awkward.  Provide an easier way
> >>>>>> for reflective code to access such non-exported types.
> >>>>>> [1]
> >>>>>>
> >>>>>> Proposal --------
> >>>>>>
> >>>>>> Extend the language of module declarations so that a
> >>>>>> package can be declared to be exported at run time but
> >>>>>> not at compile time.  This is, roughly, the dual of the
> >>>>>> `requires static` construct proposed for
> >>>>>> #CompileTimeDependences, hence we propose to introduce a
> >>>>>> new modifier, `dynamic`, for use on the `exports`
> >>>>>> directive.  It has the following meanings:
> >>>>>>
> >>>>>> - At compile time, `exports dynamic P` does not cause
> >>>>>> the package `P` to be exported, though it does require
> >>>>>> `P` to be a package defined in the module.
> >>>>>>
> >>>>>> - In phases after compile time, `exports dynamic P`
> >>>>>> behaves in
> >>>> exactly
> >>>>>> the same way as `exports P`.  It therefore takes part
> >>>>>> fully in resolution and configuration, and is subject to
> >>>>>> the same
> >>>> consistency
> >>>>>> constraints as normally-exported packages (e.g., no
> >>>>>> split
> >>>> packages).
> >>>>>>
> >>>>>> Thus a module declaration of the form
> >>>>>>
> >>>>>> module com.foo.app { requires hibernate.core; requires
> >>>>>> hibernate.entitymanager; exports dynamic
> >>>>>> com.foo.app.model; }
> >>>>>>
> >>>>>> makes the types in the `com.foo.app.model` package
> >>>>>> accessible at run
> >>>> time
> >>>>>> but not at compile time.  In combination with the
> >>>>>> earlier change to enable #ReflectionWithoutReadability
> >>>>>> [2] this means that frameworks
> >>>> that
> >>>>>> today use core reflection to manipulate user classes at
> >>>>>> run time are
> >>>> more
> >>>>>> likely to work out-of-the-box, without change, as
> >>>>>> automatic modules.
> >>>> If
> >>>>>> the `com.foo.app.model` package in this example includes
> >>>>>> entity classes to be managed by Hibernate then the
> >>>>>> framework will be able to access
> >>>> them
> >>>>>> without further ado, but under normal circumstances an
> >>>>>> attempt to
> >>>> compile
> >>>>>> code that refers directly to those classes will fail.
> >>>>>>
> >>>>>> The `dynamic` modifier can be applied to both unqualified
> >>>>>> and qualified `exports` directives, though the caveats on
> >>>>>> using qualified exports [3] still apply.
> >>>>>>
> >>>>>> Notes -----
> >>>>>>
> >>>>>> - To access a non-public member in a
> >>>>>> dynamically-exported package you must still invoke the
> >>>>>> appropriate `setAccessible` method, just as you do today
> >>>>>> to access non-public members in an exported package.
> >>>>>>
> >>>>>> - This proposal is similar to Rémi's suggestion to add a
> >>>>>> way to
> >>>> export
> >>>>>> a package only for reflection, still requiring that
> >>>>>> `setAccessible` be invoked [4].  This proposal differs
> >>>>>> in that it does not require `setAccessible` to be invoked
> >>>>>> to access public members in a dynamically-exported
> >>>>>> package.
> >>>>>>
> >>>>>> - This proposal primarily addresses "friendly" uses of
> >>>>>> reflection,
> >>>> such
> >>>>>> as dependency injection and persistence, in which the
> >>>>>> author of a module knows in advance that one or more
> >>>>>> packages must be exported
> >>>> at
> >>>>>> run time for reflective access by frameworks.  This
> >>>>>> proposal is
> >>>> also
> >>>>>> convenient for frameworks that generate code at run
> >>>>>> time, since the constant pool of a generated class file
> >>>>>> can include static
> >>>> references
> >>>>>> to types in dynamically exported packages.
> >>>>>>
> >>>>>> - This proposal opens the door to a category of
> >>>>>> second-class APIs.
> >>>> If
> >>>>>> a package is exported dynamically then you can still
> >>>>>> compile code that refers to types in the package, by
> >>>>>> using `-XaddExports` or its equivalent at compile time,
> >>>>>> and it will work as expected at run
> >>>> time.
> >>>>>> It thus may be useful to use `exports dynamic` for
> >>>>>> packages that contain legacy APIs whose use is strongly
> >>>>>> discouraged, e.g., those
> >>>> in
> >>>>>> the `jdk.unsupported` module of the reference
> >>>>>> implementation,
> >>>> thereby
> >>>>>> forcing anyone who wants to compile against them to go
> >>>>>> to the
> >>>> trouble
> >>>>>> of using `-XaddExports`.
> >>>>>>
> >>>>>> - Intrusive access to arbitrary packages of arbitrary
> >>>>>> modules by,
> >>>> e.g.,
> >>>>>> debugging tools, will still require the use of sharp
> >>>>>> knives such as the `-XaddExports` command-line option or
> >>>>>> its equivalent, or JVM
> >>>> TI.
> >>>>>>
> >>>>>> - Using the `-XaddExports` option or its equivalent
> >>>>>> remains awkward, and sometimes it's the only way out.  To
> >>>>>> ease migration I think
> >>>> it's
> >>>>>> worth considering some way for an application packaged as
> >>>>>> a JAR
> >>>> file
> >>>>>> to include such options in its `MANIFEST.MF` file, as
> >>>>>> suggested by Simon Nash [5].  I'll create a separate
> >>>>>> issue for that.
> >>>>>>
> >>>>>> - New kinds of resolution failures are possible.  If
> >>>>>> module `A` requires `B`, and `B` requires `C`, then if
> >>>>>> `B` and `C` both
> >>>> declare
> >>>>>> some package `P` to be exported dynamically, then a
> >>>>>> split-package error will be reported.  This may
> >>>>>> surprise, since the packages are just internal
> >>>>>> implementation details that were exported dynamically so
> >>>>>> that some framework could access their types.  This is
> >>>>>> not, however, all that different from the kinds of
> >>>>>> resolution and layer-creation failures that are already
> >>>>>> possible due to package collisions.  It's unlikely to
> >>>>>> happen all that often in practice, and the fix is
> >>>>>> straightforward: Just use reverse-DNS or otherwise
> >>>>>> sufficiently-unique package names, as most people already
> >>>>>> do. It is worth exploring whether javac can detect at
> >>>>>> least some of these kinds of collisions and issue
> >>>>>> appropriate warnings.
> >>>>>
> >>>>> I would like to agree on that proposal only on the ground
> >>>>> that it
> >>>> requires
> >>>>> to change the classfile format thus allows in the same time
> >>>>> to fix the issues of the current classfile spec (encoding
> >>>>> of the name module as a Class instead of UTF8, wrong flag
> >>>>> for requires public), but i'm not sure this proposal
> >>>>> support it's own weight.
> >>>>>
> >>>>> Without dynamic, a 'classical' export works the same way
> >>>>> at runtime so adding 'dynamic' to an export only restrict
> >>>>> the usage at compile time. Given that we can already
> >>>>> selectively export packages to some chosen modules which
> >>>>> hide them for the rest of the other modules at compile
> >>>> time,
> >>>>> adding dynamic to the spec adds little value.
> >>>>>
> >>>>> And it has several drawbacks, the main one is that it
> >>>>> weaken the compiletime/runtime fidelity story, with a
> >>>>> dynamic export the runtime
> >>>> world
> >>>>> is not the compiletime world anymore. The main motivation
> >>>>> to add dynamic seems to be for backward compatibility
> >>>>> during the transition between the classpath world and the
> >>>>> full module
> >>>> world,
> >>>>> for several other cases, we use the command line for that,
> >>>>> i don't see
> >>>> why
> >>>>> this case is so special that it should not work like the
> >>>>> other compatibility issues.
> >>>>>
> >>>>> Moreover, an agent, an annotation processor and a jlink
> >>>>> plugin are all able to implement the same trick, add export
> >>>>> after the compilation and before the runtime, so i don't
> >>>>> think it's a good idea to change the spec for that.
> >>>>>
> >>>>> Rémi
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>
> >>>> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccess
> >>
> >>>>
> ToNonExportedTypes
> >>>>>>
> >>>>
> >> [2]
> >>>>>>
> >>>>>
> >>>> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectionWithou
> >>
> >>>>
> tReadability
> >>>>>>
> >>>>
> >> [3]
> >>>>> http://openjdk.java.net/projects/jigsaw/spec/sotms/#qualified-export
> >>
> >>>>>
> s
> >>>>>>
> >>>>>
> >> [4]
> >>>>>>
> >>>>>
> >>>> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-Decembe
> >>
> >>>>
> r/000205.html
> >>>>>>
> >>>>
> >> [5]
> >>>>>>
> >>>>>
> >>>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/00574
> >>
> >>>>
> 5.html
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>>
> >> --
> >>
> >> PGP Key:
> >> http://keys.gnupg.net/pks/lookup?op=vindex&search=0xCA3BAD2E9CCCD509
> >>
> >>
> >>
> Web:
> >> http://codefx.org a blog about software development
> >> http://do-foss.de Free and Open Source Software for the City of
> >> Dortmund
> >>
> >> Twitter: https://twitter.com/nipafx
> >>
> >
>
> --
>
> PGP Key:
>     http://keys.gnupg.net/pks/lookup?op=vindex&search=0xCA3BAD2E9CCCD509
>
> Web:
>     http://codefx.org
>         a blog about software development
>     http://do-foss.de
>         Free and Open Source Software for the City of Dortmund
>
> Twitter:
>     https://twitter.com/nipafx
>
>