From martinrb at google.com  Mon Dec  5 20:13:16 2016
From: martinrb at google.com (Martin Buchholz)
Date: Mon, 5 Dec 2016 12:13:16 -0800
Subject: Refresh of module system coming to jdk9/dev soon
In-Reply-To: <13961035-d87c-228e-853e-2a33617b90e3@oracle.com>
References: <13961035-d87c-228e-853e-2a33617b90e3@oracle.com>
Message-ID: <CA+kOe0-kZT3cCLUrtfe+puLkHhPwHgQGF8bJtDHB=j9v_nUZ+g@mail.gmail.com>

Whitebox testing is the most obvious example of code that has a good excuse
for poking inside implementation details.
For running jsr166 tests, I'm happy to report that the following seems to
work:

  <jvmarg value="-Xbootclasspath/p:@{classes}" unless:set="modules"/>
  <jvmarg line="--patch-module java.base=@{classes}" if:set="modules"/>
+ <jvmarg line="--add-opens java.base/java.util=ALL-UNNAMED"
if:set="modules"/>
+ <jvmarg line="--add-opens java.base/java.util.concurrent=ALL-UNNAMED"
if:set="modules"/>
+ <jvmarg line="--add-opens
java.base/java.util.concurrent.atomic=ALL-UNNAMED" if:set="modules"/>
+ <jvmarg line="--add-opens
java.base/java.util.concurrent.locks=ALL-UNNAMED" if:set="modules"/>

It would be nice if there was explicit guidance for authors of tooling
software like test harnesses.  jtreg has some magic in JTRegModuleHelper
that calls Module.implAddOpens reflectively, but that's not going to work
for ordinary software, and there's no cookbook recipe for how to do that.
 jtreg has the advantage that it is already spawning other java processes,
and so can inject arbitrary command line flags.

It would be nice if the production software could define test-only methods
(think "checkInvariants()") that could be invoked without having to resort
to reflection.  Google has some use-this-method-in-tests-only tooling that
uses annotations, but the enforcement is external to the jdk.  I keep
hoping that module systems would somehow work at the method level instead
of the package level, because we already have visibility keywords like
private at the method level.

Giving access to internals to legitimate tools while preventing other uses
seems like a Hard Problem.  For tools like debuggers or heap analyzers that
really want to see everything, there will be a strong desire to "disable
all module checks".  Or maybe the right way is for users to patch them into
java.base?  Guidance needed.


On Wed, Nov 30, 2016 at 4:39 AM, Alan Bateman <Alan.Bateman at oracle.com>
wrote:

> Just a heads-up that the code review and cleanup for an update of the
> module system is currently underway on jigsaw-dev [1] with a view to
> pushing the changes to jdk9/dev soon, maybe for jdk-9+148 that will promote
> next week.
>
> For those that have been trying out modules with regular JDK 9 builds then
> be aware that `requires public` changes to `requires transitive`. In
> addition, the binary representation of the module declaration
> (module-info.class) has changed so that you need to recompile any modules
> that were compiled with previous JDK 9 builds.
>
> This refresh includes a disruptive change that is important to understand.
> As things stand today in JDK 9 then you use setAccessible to break into
> non-public elements of any type in exported packages (you can hack into
> private fields of any type in java.util example). However, it cannot be
> used to break into any type in non-exported package (you can't break into
> types in jdk.internal.misc or sun.security.x509 for example). The current
> specified behavior was a compromise for the initial integration of the
> module system. It is of course not very satisfactory, hence the
> #AwkwardStrongEncapsulation issue [2] on the JSR 376 issues list. With the
> updated proposal in the JSR, this refresh changes setAccessible further so
> that it cannot be used to break into non-public types, or non-public
> elements of public types, in exported packages. Code that uses
> setAccessible to hack into the private constructor of
> java.lang.invoke.MethodHandles.Lookup will be disappointed for example.
>
> This change will expose hacks in many existing libraries and tools (as
> happened when setAccessible was changed to prevent it being used to break
> into sun.* packages). It will not be popular. As a workaround then a new
> command line option `--add-opens` can be used to open specific packages for
> "deep reflection". For example, a really popular build tool fails with this
> refresh because it uses setAccessible + core reflection to hack into a
> private field of an unmodifiable collection so that it can mutate it,
> facepalm! This code will continue to work as before when run with
> `--add-opens java.base/java.util=ALL-UNNAMED` to open the package
> java.util in module java.base to "all unnamed modules" (think class path).
>
> As I said, this change will not be popular but please bear with it until
> the extent of the issues uncovered is more widely understood. We need all
> the help we can get to identify issues and get them reported (and hopefully
> fixed) by the libraries and tools with the technical debt (and we expect a
> lot of it will be technical debt). For those working on OpenJDK and the JDK
> 9 project then it's very possible that some of these issues will redirect
> to the JDK as requests for new APIs or hooks (and I assume will need to be
> looked at on a case-by-case basis).
>
> Any help reporting issues to popular tools and libraries would be
> appreciated. A debugging aid that is useful to identify issues is to run
> with -Dsun.reflect.debugModuleAccessChecks=true to get a stack trace when
> setAccessible fails, this is particularly useful when code swallows
> exceptions without any logging.
>
> One final point, and only interesting to those working in OpenJDK, is that
> this refresh will require upgrading your build of jtreg. The changes to
> work with this refresh have been in the code-tools/jtreg repository for
> some time. The TEST.ROOT in each repository has been updated to require the
> new version.
>
> -Alan
>
> [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-Novem
> ber/010249.html
> [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#Awkward
> StrongEncapsulation
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20161205/8755f8ef/attachment.html>

From Alan.Bateman at oracle.com  Tue Dec  6 12:22:43 2016
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Tue, 6 Dec 2016 12:22:43 +0000
Subject: Refresh of module system coming to jdk9/dev soon
In-Reply-To: <CA+kOe0-kZT3cCLUrtfe+puLkHhPwHgQGF8bJtDHB=j9v_nUZ+g@mail.gmail.com>
References: <13961035-d87c-228e-853e-2a33617b90e3@oracle.com>
	<CA+kOe0-kZT3cCLUrtfe+puLkHhPwHgQGF8bJtDHB=j9v_nUZ+g@mail.gmail.com>
Message-ID: <86df173f-21f3-22c0-320a-c3cba3b35310@oracle.com>

On 05/12/2016 20:13, Martin Buchholz wrote:

> Whitebox testing is the most obvious example of code that has a good 
> excuse for poking inside implementation details.
> For running jsr166 tests, I'm happy to report that the following seems 
> to work:
>
> <jvmarg value="-Xbootclasspath/p:@{classes}" unless:set="modules"/>
> <jvmarg line="--patch-module java.base=@{classes}" if:set="modules"/>
> +<jvmarg line="--add-opens java.base/java.util=ALL-UNNAMED" 
> if:set="modules"/>
> +<jvmarg line="--add-opens java.base/java.util.concurrent=ALL-UNNAMED" 
> if:set="modules"/>
> +<jvmarg line="--add-opens 
> java.base/java.util.concurrent.atomic=ALL-UNNAMED" if:set="modules"/>
> +<jvmarg line="--add-opens 
> java.base/java.util.concurrent.locks=ALL-UNNAMED" if:set="modules"/>
>
> It would be nice if there was explicit guidance for authors of tooling 
> software like test harnesses.
There have been a number of threads on jigsaw-dev on this topic. 
Whitebox tests that are in the same package as the API under test (by 
convention or because they are testing package-private methods) can be 
compiled and run "as if" they are part of the module. This is done with 
the --patch-module option, like what you are doing to override the 
jsr166 classes in java.base. Code in a module can do "deep reflection" 
on any element of any type in its own module so the tests shouldn't have 
any issue when they are in the module. That said, there might some 
additional command line options needed to allow the test runner invoke 
the tests (where the tests are added to the module in a non-exported 
package) or where the tests links to the supporting test library 
(probably JUnit here). We've worked with many scenarios where everything 
is modules or where there is a mix of modules and class path, it can all 
be made to work. In time then I hope the test runners will make this 
easier but there isn't anything specifically written up yet.

>
> It would be nice if the production software could define test-only 
> methods (think "checkInvariants()") that could be invoked without 
> having to resort to reflection.  Google has some 
> use-this-method-in-tests-only tooling that uses annotations, but the 
> enforcement is external to the jdk.  I keep hoping that module systems 
> would somehow work at the method level instead of the package level, 
> because we already have visibility keywords like private at the method 
> level.
If the checkInvariant methods aren't public then the packages that they 
are in will needed to be opened to whoever calls them. For the scenario 
then qualified exports or qualified opens can be used to export and/or 
open specific packages to test modules.

>
> Giving access to internals to legitimate tools while preventing other 
> uses seems like a Hard Problem.  For tools like debuggers or heap 
> analyzers that really want to see everything, there will be a strong 
> desire to "disable all module checks".  Or maybe the right way is for 
> users to patch them into java.base?  Guidance needed.
We updated the debugger support and the tools APIs to support modules 
some time ago. I'm not sure if we have a specific issue in mind but the 
debugger support involves JVM TI and JNI on the backend so there isn't 
an issue with accessibility. Heap analyzers that are based on the tool 
APIs should be fine too, as should tools and agents based on the 
instrumentation API (it's even possible for agents that don't know 
anything about modules to instrument code in modules).

-Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20161206/ab79c53a/attachment.html>

From martinrb at google.com  Thu Dec  8 03:32:48 2016
From: martinrb at google.com (Martin Buchholz)
Date: Wed, 7 Dec 2016 19:32:48 -0800
Subject: Refresh of module system coming to jdk9/dev soon
In-Reply-To: <86df173f-21f3-22c0-320a-c3cba3b35310@oracle.com>
References: <13961035-d87c-228e-853e-2a33617b90e3@oracle.com>
	<CA+kOe0-kZT3cCLUrtfe+puLkHhPwHgQGF8bJtDHB=j9v_nUZ+g@mail.gmail.com>
	<86df173f-21f3-22c0-320a-c3cba3b35310@oracle.com>
Message-ID: <CA+kOe08Z5b3esaKEwtbH0RATE4VbJGX1mVZvDJCwKJLWpvWTfQ@mail.gmail.com>

I plan to keep on using reflection to access implementation details in
tests, using --add-opens as necessary.  That does require having control
over the commandline, or using a module-aware test framework that takes
care of that for you (like jtreg).

We still don't have a solution to the inheritDoc problem - how to inherit
doc comments from javadoc in superclasses in modules.

The tools I was thinking of that might need to be jigsaw aware were jol and
jmh.


On Tue, Dec 6, 2016 at 4:22 AM, Alan Bateman <Alan.Bateman at oracle.com>
wrote:

> On 05/12/2016 20:13, Martin Buchholz wrote:
>
> Whitebox testing is the most obvious example of code that has a good
> excuse for poking inside implementation details.
> For running jsr166 tests, I'm happy to report that the following seems to
> work:
>
>   <jvmarg value="-Xbootclasspath/p:@{classes}" unless:set="modules"/>
>   <jvmarg line="--patch-module java.base=@{classes}" if:set="modules"/>
> + <jvmarg line="--add-opens java.base/java.util=ALL-UNNAMED"
> if:set="modules"/>
> + <jvmarg line="--add-opens java.base/java.util.concurrent=ALL-UNNAMED"
> if:set="modules"/>
> + <jvmarg line="--add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED"
> if:set="modules"/>
> + <jvmarg line="--add-opens java.base/java.util.concurrent.locks=ALL-UNNAMED"
> if:set="modules"/>
>
> It would be nice if there was explicit guidance for authors of tooling
> software like test harnesses.
>
> There have been a number of threads on jigsaw-dev on this topic. Whitebox
> tests that are in the same package as the API under test (by convention or
> because they are testing package-private methods) can be compiled and run
> "as if" they are part of the module. This is done with the --patch-module
> option, like what you are doing to override the jsr166 classes in
> java.base. Code in a module can do "deep reflection" on any element of any
> type in its own module so the tests shouldn't have any issue when they are
> in the module. That said, there might some additional command line options
> needed to allow the test runner invoke the tests (where the tests are added
> to the module in a non-exported package) or where the tests links to the
> supporting test library (probably JUnit here). We've worked with many
> scenarios where everything is modules or where there is a mix of modules
> and class path, it can all be made to work. In time then I hope the test
> runners will make this easier but there isn't anything specifically written
> up yet.
>
>
> It would be nice if the production software could define test-only methods
> (think "checkInvariants()") that could be invoked without having to resort
> to reflection.  Google has some use-this-method-in-tests-only tooling that
> uses annotations, but the enforcement is external to the jdk.  I keep
> hoping that module systems would somehow work at the method level instead
> of the package level, because we already have visibility keywords like
> private at the method level.
>
> If the checkInvariant methods aren't public then the packages that they
> are in will needed to be opened to whoever calls them. For the scenario
> then qualified exports or qualified opens can be used to export and/or open
> specific packages to test modules.
>
>
> Giving access to internals to legitimate tools while preventing other uses
> seems like a Hard Problem.  For tools like debuggers or heap analyzers that
> really want to see everything, there will be a strong desire to "disable
> all module checks".  Or maybe the right way is for users to patch them into
> java.base?  Guidance needed.
>
> We updated the debugger support and the tools APIs to support modules some
> time ago. I'm not sure if we have a specific issue in mind but the debugger
> support involves JVM TI and JNI on the backend so there isn't an issue with
> accessibility. Heap analyzers that are based on the tool APIs should be
> fine too, as should tools and agents based on the instrumentation API (it's
> even possible for agents that don't know anything about modules to
> instrument code in modules).
>
> -Alan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20161207/918459e1/attachment-0001.html>

From Alan.Bateman at oracle.com  Thu Dec  8 08:27:56 2016
From: Alan.Bateman at oracle.com (Alan Bateman)
Date: Thu, 8 Dec 2016 08:27:56 +0000
Subject: Refresh of module system coming to jdk9/dev soon
In-Reply-To: <CA+kOe08Z5b3esaKEwtbH0RATE4VbJGX1mVZvDJCwKJLWpvWTfQ@mail.gmail.com>
References: <13961035-d87c-228e-853e-2a33617b90e3@oracle.com>
	<CA+kOe0-kZT3cCLUrtfe+puLkHhPwHgQGF8bJtDHB=j9v_nUZ+g@mail.gmail.com>
	<86df173f-21f3-22c0-320a-c3cba3b35310@oracle.com>
	<CA+kOe08Z5b3esaKEwtbH0RATE4VbJGX1mVZvDJCwKJLWpvWTfQ@mail.gmail.com>
Message-ID: <6159506e-d0f7-cfa2-d393-7e661e888eee@oracle.com>

On 08/12/2016 03:32, Martin Buchholz wrote:

> I plan to keep on using reflection to access implementation details in 
> tests, using --add-opens as necessary.  That does require having 
> control over the commandline, or using a module-aware test framework 
> that takes care of that for you (like jtreg).
That should work fine, or as I said, if the tests are compiled and run 
"as if" they are in the module being tested then they will be able to do 
deep reflective on the internal/implementation without any needed to 
open the packages.

>
> We still don't have a solution to the inheritDoc problem - how to 
> inherit doc comments from javadoc in superclasses in modules.
I assume you'll bring this up on javadoc-dev.

>
> The tools I was thinking of that might need to be jigsaw aware were 
> jol and jmh.
I assume any issues can be brought to jol-dev or jmh-dev. I recall there 
was an issue with JMH a few months ago related to how it detects the 
console encoding, I believe Claes or Aleksey has put in a fix or 
workaround for this. For JOL then it uses SA which is not an exported 
API, I believe it launches with --add-exports options to export the 
packages containing the SA classes.

-Alan