Proposal: #ReflectiveAccessToNonExportedTypes: `exports dynamic`

Nicolai Parlog nipa at codefx.org
Tue Jul 5 16:05:34 UTC 2016 

 Hi Rafael.

> As for JUnit 4 or Unsafe, reflection is used because of an actual
> need to interact with these libraries/APIs.

Yes, definitely. I hope I didn't come across as "these stupid Unsafe
users didn't know what they were doing!".

> As for JUnit, it is mainly IDE vendors that needed to access
> internals in order to display unit testing results and to extract
> details about testing results. By making this extraction more
> awkward, this would not have changed the fact that IDE-vendors want
> to interact with the internal APIs. It  would only have been more
> difficult to do.

Alternatively, they could've improved JUnit 4's API, leading to the
following (ideally):

* faster improvements of JUnit 4 because less barriers to refactoring
* one less need for a total rewrite (!)
* no DRY across tools, leading to more time for other improvements
* an API for new tools to build on (enabling innovation),
  which would otherwise have to repeat the same time-intensive hacks

And the only downside being that JUnit integration would initially have
been a little slower, which I think is a fair price to pay. I am
convinced that JUnit is a perfect example where the ease, with which
reflection allows to break encapsulation, showed immense detrimental
effects with no clear advantage.

The argument about Unsafe is pretty much the same; with a critical
difference: It is much harder and much slower to move the JDK along than
it is to do the some to "some open source project". This considerably
weakens my argument, of course, maybe making your point for you. Some
innovations may not have happened were it not for said weak encapsulation.

> Also, as a thought experiment, let us say accessing internal APIs had
> been impossible in the past: we would have missed out on Hadoop,
> Cassandra, Mockito, Spark; and only in 2016 we could start running
> JUnit from our IDE.

But what about the command line flags ("addexports" et al)? They
would've made adoption of these innovations slower but if they are
addressing a real need, wouldn't that be worth it? And much like in the
case of JUnit would it not also increase pressure on the JDK developers
to create a supported alternative?

Honestly, I just don't know. :)

Regarding the rest of your arguments. Many of them also apply to
visibility modifiers. As so often it's a trade-off (duh). And as so
often we only have past data (warped by what was possible) and
hypothesis (warped, amongst other things, by what we work on) to decide
what to do. So I think it is important to argue both sides of it and see
what comes out in the end. :)

I, for one, think that Carthago... eh that reflection should not be
allowed to break encapsulation (without command line flags).

 so long ... Nicolai



On 01.07.2016 11:15, Rafael Winterhalter wrote:
> As for JUnit 4 or Unsafe, reflection is used because of an actual need to
> interact with these libraries/APIs.
> 
> As for JUnit, it is mainly IDE vendors that needed to access internals in
> order to display unit testing results and to extract details about testing
> results. By making this extraction more awkward, this would not have
> changed the fact that IDE-vendors want to interact with the internal APIs.
> It  would only have been more difficult to do. In my eyes, limiting
> reflection is fighting the symptoms, not the cause. For example, JUnit 5
> has adressed this issue by implementing a public API for extracting such
> information what immediately solved the current problem. I do not think
> anybody prefers reflection over an official API if the latter one satisfies
> all needs.
> 
> The same goes for Unsafe. I do not think anybody likes to use Unsafe if
> there is a better alternative. The Unsafe API is awkward and - obviously -
> unsafe. And despite the implementors attempt to shield the API to bootstrap
> classes, people found their ways around because there has not been another
> way to make certain things work. I am sure, the more the JVM offers
> adequate alternatives, people will migrate happily.
> 
> Also, as a thought experiment, let us say accessing internal APIs had been
> impossible in the past: we would have missed out on Hadoop, Cassandra,
> Mockito, Spark; and only in 2016 we could start running JUnit from our IDE.
> I think we can all agree that this is not the outcome we would have wanted.
> Reflection - despite it breaking encapsulation - has done more good then
> bad.
> 
> As for the argument that required dynamic exports should be documented;
> ideally yes. Equally, APIs should be documented but many librararies fail
> to offer good documentation, especially in the open-source space where time
> is sparse. I already see myself answering questions to puzzled junior
> developers that know Java for a few months that "this is required because
> this library uses reflection" where they do not even know what the latter
> is. Effectively, building this barrier makes Java more difficult to use and
> I am convinced that many people will adapt adding dynamic exports for any
> package as some form of "best practice" to get on with their lifes. I will
> also bet that someone will deliver some "auto-export Maven plugin" to
> automate this "best practive".
> 
> Finally, I do not believe the service loader API offers an adequate
> alternative to many use-cases of reflection (while it can perfectly
> substitute some of them). Many state-of-the art libraries actually
> intentionally offer POJO approaches without requiring their users to create
> interfaces for any shared interaction. I do not think that a gentle push
> into a direction that contradicts most existing Java libraries would be a
> taken well by the people implementing this software and in the end, the
> module system will be as successful as its adoption. I rather see people
> finding new ways around restrictions to keep things simple (for themselves).
> 
> For all these reasons I do not think that it is a good idea to limit
> reflection to exported packages.
> 
> 2016-07-01 9:27 GMT+02:00 Nicolai Parlog <nipa at codefx.org>:
> 
>>  Hi!
>>
>>> I do not think that shielding internals is a problem with regards
>>> to reflection.
>>
>> I think it is. I've not been emerged in the community for very long so
>> I know only two examples (but still): Unsafe and JUnit 4. In both
>> cases internals were accessed via reflection. And in both cases
>> dependencies became so ubiquitous that changing any details became
>> impossible for the maintainers.
>>
>> I think the outcry when Unsafe was planned to go away and the fact
>> that JUnit 4 had to be rewritten because evolution was effectively
>> prevented by unwanted dependencies make my point here.
>>
>>> I do neither think that reflection only interests internals.
>>
>> I agree. Not "only" but "all too often", though.
>>
>>> For example, maybe you use Hibernate for your persistance layer
>>> without exposing those DTOs from your module.
>>
>> Why would I do that?! This is exactly where reflection got us used to
>> sloppy architecture. If a tool uses my DTOs to implement my
>> persistence layer, then it looks like those DTOs are part of my API.
>> Maybe not public to everyone but qualified and dynamic exports give me
>> tools to reduce visibility.
>>
>>> I do not care about how Hibernate works as long as it does work.
>>> Almost any library uses reflection and I would not want to dig
>>> into implementation details to find out how to set up my dynamic
>>> exports.
>>
>> Maybe I'm missing something here but to me it looks like "export to a
>> module whatever that module is supposed to interact with" is a very
>> sensible default that should cover a vast majority of uses cases, no?
>>
>>         I have DTOs && Hibernate needs DTOs -> export DTOs to Hibernate.
>>
>> Why would I look at the implementation? And if there were non-obvious
>> scenarios they should be properly documented so I can react accordingly.
>>
>>> As a library implementor, you aim for the biggest convenience.
>>
>> Maybe we should aim for clarity and well-defined interactions instead.
>>
>>  so long ... Nicolai
>>
>>
>>
>> On 01.07.2016 00:20, Rafael Winterhalter wrote:
>>> I disagree on two ends here.
>>>
>>> I do not think that shielding internals is a problem with regards
>>> to reflection. You want to shield internals in order to not break
>>> dependants by prohibiting their existance. With reflection, you do
>>> however typically react to the actual implementation by discovering
>>> it at runtime. If your use pattern was fixed, you would always
>>> require a static dependency over reflection in the first place. I
>>> do neither think that reflection only interests internals. For
>>> example, maybe you use Hibernate for your persistance layer without
>>> exposing those DTOs from your module. The problem is that you
>>> invert responsibilities. I do not care about how Hibernate works as
>>> long as it does work. Almost any library uses reflection and I
>>> would not want to dig into implementation details to find out how
>>> to set up my dynamic exports. I want to require a module and
>>> hopefully the compiler tells me about anything that's wrong.
>>> Reflection access errors would only be an awkward side effect that
>>> can be avoided by not applying modules alltogether. I think this
>>> limitation would therefore hurt Java 9 adoption.
>>>
>>> The second problem I have is the issue of attempting to fix an old
>>> regret. From the perspective of Hibernate, it would be easiest to
>>> not attempt modules as their introduction would require its users
>>> to apply additional configuration. As a library implementor, you
>>> aim for the biggest convenience. Adding this restriction neither
>>> adds additional security where a security manager should be used. I
>>> again think this would hurt adoption.
>>>
>>> I agree with Stephen's argument even though I do not think that
>>> forbidding dynamic access would be a feature anybody would use so
>>> it should not be added. Rather, something should be integrated in
>>> the security manager, if this is considered necessary.
>>>
>>> Cheers, Rafael
>>>
>>> Am 30.06.2016 11:37 nachm. schrieb "Nicolai Parlog"
>>> <nipa at codefx.org>:
>>>>
>>>> Hi!
>>>>
>>>>> In my opinion, reflection should not apply any module-boundary
>>>>> checks.
>>>>
>>>> Maybe this list is not a representative selection of Java
>>>> developers, maybe I'm just the only one thinking this but, to
>>>> paraphrase:
>>>>
>>>> Reflection over internals must die in a fire!
>>>>
>>>> Yes, yes, tools to great things, we need to see everything, ...
>>>> Still, I think that modules need a safe space that they can use
>>>> without anybody intruding. Maybe I'm a hardliner but I already
>>>> view the command line parameters with suspicion.
>>>>
>>>> In that light, I like the "dynamic" proposal.
>>>>
>>>> Not the name, though. Having the symmetric pair "static vs
>>>> dynamic" looks nice but it takes a lot of words to explain why
>>>> exactly these terms apply here. We seem to be looking for
>>>> qualifiers for "requires" and "exports". Maybe just using other
>>>> terms solves the problem?
>>>>
>>>> requires & consumes exports & contains
>>>>
>>>> so long ... Nicolai
>>>>
>>>>
>>>>
>>>> On 30.06.2016 01:23, Rafael Winterhalter wrote:
>>>>> I also agree here that the use of reflection should not require
>>>>> any form of explicit permission by the user of a library. In a
>>>>> way, this ties an implementation detail (reflection in this
>>>>> case) to the usage of the library. The library cannot work
>>>>> without the explicit configuration by a user and this only adds
>>>>> boilerplate to using Hibernate.
>>>>>
>>>>> Unfortunately, many Java developers still do not know much
>>>>> about reflection or about the inner workings of many libraries
>>>>> in the market. If a user needed to explicitly export packages,
>>>>> this might just lead to people "over exporting" their packages
>>>>> just to "make Hibernate work".
>>>>>
>>>>> In my opinion, reflection should not apply any module-boundary
>>>>> checks.
>>>>>
>>>>> Best regards, Rafael
>>>>>
>>>>> 2016-06-29 20:13 GMT+02:00 Paul Benedict
>>>>> <pbenedict at apache.org>:
>>>>>
>>>>>> Replying to "observers" list...
>>>>>>
>>>>>> I think the main problem here is that core reflection has
>>>>>> become too restrictive. Reflection should be "magical" and
>>>>>> have no knowledge of any kind of module and/or visibility
>>>>>> boundaries. If you really want to control what can be
>>>>>> reflected upon, that should be done via the Policy file of
>>>>>> the Security Manager.
>>>>>>
>>>>>> This proposal is trying to hack out-of-the-box created by
>>>>>> too strict of a design. It is very awkward to denote which
>>>>>> packages must be reflectable ahead of time --- how can you
>>>>>> figure that out?? The consumers who are doing the reflection
>>>>>> are innumerable. I'd go as far to say it's it's impossible to
>>>>>> devise a preemptive comprehensive strategy that may satisfy
>>>>>> the kinds of tooling in the wild.
>>>>>>
>>>>>> It's best just to open up reflection so it can continue
>>>>>> doing what it always has done well: let me see anything I
>>>>>> want unless the Security Manager says no.
>>>>>>
>>>>>> Cheers, Paul
>>>>>>
>>>>>> On Wed, Jun 29, 2016 at 3:51 AM, Remi Forax
>>>>>> <forax at univ-mlv.fr> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----- Mail original -----
>>>>>>>> De: "Mark Reinhold" <mark.reinhold at oracle.com> À:
>>>>>>>> jpms-spec-experts at openjdk.java.net Envoyé: Mardi 28 Juin
>>>>>>>> 2016 23:18:15 Objet: Proposal:
>>>>>>>> #ReflectiveAccessToNonExportedTypes: `exports dynamic`
>>>>>>>>
>>>>>>>> Issue summary -------------
>>>>>>>>
>>>>>>>> #ReflectiveAccessToNonExportedTypes --- Some kinds of
>>>>>>>> framework libraries require reflective access to members
>>>>>>>> of the non-exported types of other modules; examples
>>>>>>>> include dependency injection
>>>>>> (Guice),
>>>>>>>> persistence (JPA), debugging tools, code-automation
>>>>>>>> tools, and serialization (XStream).  In some cases the
>>>>>>>> particular library to be used is not known until run time
>>>>>>>> (e.g., Hibernate and EclipseLink
>>>>>> both
>>>>>>>> implement JPA).  This capability is also sometimes used
>>>>>>>> to work
>>>>>> around
>>>>>>>> bugs in unchangeable code.  Access to non-exported
>>>>>>>> packages can, at present, only be done via command-line
>>>>>>>> flags, which is extremely awkward.  Provide an easier way
>>>>>>>> for reflective code to access such non-exported types.
>>>>>>>> [1]
>>>>>>>>
>>>>>>>> Proposal --------
>>>>>>>>
>>>>>>>> Extend the language of module declarations so that a
>>>>>>>> package can be declared to be exported at run time but
>>>>>>>> not at compile time.  This is, roughly, the dual of the
>>>>>>>> `requires static` construct proposed for
>>>>>>>> #CompileTimeDependences, hence we propose to introduce a
>>>>>>>> new modifier, `dynamic`, for use on the `exports`
>>>>>>>> directive.  It has the following meanings:
>>>>>>>>
>>>>>>>> - At compile time, `exports dynamic P` does not cause
>>>>>>>> the package `P` to be exported, though it does require
>>>>>>>> `P` to be a package defined in the module.
>>>>>>>>
>>>>>>>> - In phases after compile time, `exports dynamic P`
>>>>>>>> behaves in
>>>>>> exactly
>>>>>>>> the same way as `exports P`.  It therefore takes part
>>>>>>>> fully in resolution and configuration, and is subject to
>>>>>>>> the same
>>>>>> consistency
>>>>>>>> constraints as normally-exported packages (e.g., no
>>>>>>>> split
>>>>>> packages).
>>>>>>>>
>>>>>>>> Thus a module declaration of the form
>>>>>>>>
>>>>>>>> module com.foo.app { requires hibernate.core; requires
>>>>>>>> hibernate.entitymanager; exports dynamic
>>>>>>>> com.foo.app.model; }
>>>>>>>>
>>>>>>>> makes the types in the `com.foo.app.model` package
>>>>>>>> accessible at run
>>>>>> time
>>>>>>>> but not at compile time.  In combination with the
>>>>>>>> earlier change to enable #ReflectionWithoutReadability
>>>>>>>> [2] this means that frameworks
>>>>>> that
>>>>>>>> today use core reflection to manipulate user classes at
>>>>>>>> run time are
>>>>>> more
>>>>>>>> likely to work out-of-the-box, without change, as
>>>>>>>> automatic modules.
>>>>>> If
>>>>>>>> the `com.foo.app.model` package in this example includes
>>>>>>>> entity classes to be managed by Hibernate then the
>>>>>>>> framework will be able to access
>>>>>> them
>>>>>>>> without further ado, but under normal circumstances an
>>>>>>>> attempt to
>>>>>> compile
>>>>>>>> code that refers directly to those classes will fail.
>>>>>>>>
>>>>>>>> The `dynamic` modifier can be applied to both unqualified
>>>>>>>> and qualified `exports` directives, though the caveats on
>>>>>>>> using qualified exports [3] still apply.
>>>>>>>>
>>>>>>>> Notes -----
>>>>>>>>
>>>>>>>> - To access a non-public member in a
>>>>>>>> dynamically-exported package you must still invoke the
>>>>>>>> appropriate `setAccessible` method, just as you do today
>>>>>>>> to access non-public members in an exported package.
>>>>>>>>
>>>>>>>> - This proposal is similar to Rémi's suggestion to add a
>>>>>>>> way to
>>>>>> export
>>>>>>>> a package only for reflection, still requiring that
>>>>>>>> `setAccessible` be invoked [4].  This proposal differs
>>>>>>>> in that it does not require `setAccessible` to be invoked
>>>>>>>> to access public members in a dynamically-exported
>>>>>>>> package.
>>>>>>>>
>>>>>>>> - This proposal primarily addresses "friendly" uses of
>>>>>>>> reflection,
>>>>>> such
>>>>>>>> as dependency injection and persistence, in which the
>>>>>>>> author of a module knows in advance that one or more
>>>>>>>> packages must be exported
>>>>>> at
>>>>>>>> run time for reflective access by frameworks.  This
>>>>>>>> proposal is
>>>>>> also
>>>>>>>> convenient for frameworks that generate code at run
>>>>>>>> time, since the constant pool of a generated class file
>>>>>>>> can include static
>>>>>> references
>>>>>>>> to types in dynamically exported packages.
>>>>>>>>
>>>>>>>> - This proposal opens the door to a category of
>>>>>>>> second-class APIs.
>>>>>> If
>>>>>>>> a package is exported dynamically then you can still
>>>>>>>> compile code that refers to types in the package, by
>>>>>>>> using `-XaddExports` or its equivalent at compile time,
>>>>>>>> and it will work as expected at run
>>>>>> time.
>>>>>>>> It thus may be useful to use `exports dynamic` for
>>>>>>>> packages that contain legacy APIs whose use is strongly
>>>>>>>> discouraged, e.g., those
>>>>>> in
>>>>>>>> the `jdk.unsupported` module of the reference
>>>>>>>> implementation,
>>>>>> thereby
>>>>>>>> forcing anyone who wants to compile against them to go
>>>>>>>> to the
>>>>>> trouble
>>>>>>>> of using `-XaddExports`.
>>>>>>>>
>>>>>>>> - Intrusive access to arbitrary packages of arbitrary
>>>>>>>> modules by,
>>>>>> e.g.,
>>>>>>>> debugging tools, will still require the use of sharp
>>>>>>>> knives such as the `-XaddExports` command-line option or
>>>>>>>> its equivalent, or JVM
>>>>>> TI.
>>>>>>>>
>>>>>>>> - Using the `-XaddExports` option or its equivalent
>>>>>>>> remains awkward, and sometimes it's the only way out.  To
>>>>>>>> ease migration I think
>>>>>> it's
>>>>>>>> worth considering some way for an application packaged as
>>>>>>>> a JAR
>>>>>> file
>>>>>>>> to include such options in its `MANIFEST.MF` file, as
>>>>>>>> suggested by Simon Nash [5].  I'll create a separate
>>>>>>>> issue for that.
>>>>>>>>
>>>>>>>> - New kinds of resolution failures are possible.  If
>>>>>>>> module `A` requires `B`, and `B` requires `C`, then if
>>>>>>>> `B` and `C` both
>>>>>> declare
>>>>>>>> some package `P` to be exported dynamically, then a
>>>>>>>> split-package error will be reported.  This may
>>>>>>>> surprise, since the packages are just internal
>>>>>>>> implementation details that were exported dynamically so
>>>>>>>> that some framework could access their types.  This is
>>>>>>>> not, however, all that different from the kinds of
>>>>>>>> resolution and layer-creation failures that are already
>>>>>>>> possible due to package collisions.  It's unlikely to
>>>>>>>> happen all that often in practice, and the fix is
>>>>>>>> straightforward: Just use reverse-DNS or otherwise
>>>>>>>> sufficiently-unique package names, as most people already
>>>>>>>> do. It is worth exploring whether javac can detect at
>>>>>>>> least some of these kinds of collisions and issue
>>>>>>>> appropriate warnings.
>>>>>>>
>>>>>>> I would like to agree on that proposal only on the ground
>>>>>>> that it
>>>>>> requires
>>>>>>> to change the classfile format thus allows in the same time
>>>>>>> to fix the issues of the current classfile spec (encoding
>>>>>>> of the name module as a Class instead of UTF8, wrong flag
>>>>>>> for requires public), but i'm not sure this proposal
>>>>>>> support it's own weight.
>>>>>>>
>>>>>>> Without dynamic, a 'classical' export works the same way
>>>>>>> at runtime so adding 'dynamic' to an export only restrict
>>>>>>> the usage at compile time. Given that we can already
>>>>>>> selectively export packages to some chosen modules which
>>>>>>> hide them for the rest of the other modules at compile
>>>>>> time,
>>>>>>> adding dynamic to the spec adds little value.
>>>>>>>
>>>>>>> And it has several drawbacks, the main one is that it
>>>>>>> weaken the compiletime/runtime fidelity story, with a
>>>>>>> dynamic export the runtime
>>>>>> world
>>>>>>> is not the compiletime world anymore. The main motivation
>>>>>>> to add dynamic seems to be for backward compatibility
>>>>>>> during the transition between the classpath world and the
>>>>>>> full module
>>>>>> world,
>>>>>>> for several other cases, we use the command line for that,
>>>>>>> i don't see
>>>>>> why
>>>>>>> this case is so special that it should not work like the
>>>>>>> other compatibility issues.
>>>>>>>
>>>>>>> Moreover, an agent, an annotation processor and a jlink
>>>>>>> plugin are all able to implement the same trick, add export
>>>>>>> after the compilation and before the runtime, so i don't
>>>>>>> think it's a good idea to change the spec for that.
>>>>>>>
>>>>>>> Rémi
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>>>>>>
>>>>>> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccess
>>>>
>>>>>>
>> ToNonExportedTypes
>>>>>>>>
>>>>>>
>>>> [2]
>>>>>>>>
>>>>>>>
>>>>>> http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectionWithou
>>>>
>>>>>>
>> tReadability
>>>>>>>>
>>>>>>
>>>> [3]
>>>>>>> http://openjdk.java.net/projects/jigsaw/spec/sotms/#qualified-export
>>>>
>>>>>>>
>> s
>>>>>>>>
>>>>>>>
>>>> [4]
>>>>>>>>
>>>>>>>
>>>>>> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-Decembe
>>>>
>>>>>>
>> r/000205.html
>>>>>>>>
>>>>>>
>>>> [5]
>>>>>>>>
>>>>>>>
>>>>>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/00574
>>>>
>>>>>>
>> 5.html
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>>
>>>> --
>>>>
>>>> PGP Key:
>>>> http://keys.gnupg.net/pks/lookup?op=vindex&search=0xCA3BAD2E9CCCD509
>>>>
>>>>
>>>>
>> Web:
>>>> http://codefx.org a blog about software development
>>>> http://do-foss.de Free and Open Source Software for the City of
>>>> Dortmund
>>>>
>>>> Twitter: https://twitter.com/nipafx
>>>>
>>>
>>
>> --
>>
>> PGP Key:
>>     http://keys.gnupg.net/pks/lookup?op=vindex&search=0xCA3BAD2E9CCCD509
>>
>> Web:
>>     http://codefx.org
>>         a blog about software development
>>     http://do-foss.de
>>         Free and Open Source Software for the City of Dortmund
>>
>> Twitter:
>>     https://twitter.com/nipafx
>>
>>
> 

-- 

PGP Key:
    http://keys.gnupg.net/pks/lookup?op=vindex&search=0xCA3BAD2E9CCCD509

Web:
    http://codefx.org
        a blog about software development
    http://do-foss.de
        Free and Open Source Software for the City of Dortmund

Twitter:
    https://twitter.com/nipafx

More information about the jpms-spec-observers mailing list