From stephan.herrmann at berlin.de Sun Apr 1 11:31:50 2018 From: stephan.herrmann at berlin.de (Stephan Herrmann) Date: Sun, 1 Apr 2018 13:31:50 +0200 Subject: Deprecation beyond the level of types Message-ID: I'm looking at how deprecation affects elements larger than types, and it seems to me that deprecation is much weaker than it could be. Packages: JLS & javadoc agree that @Deprecated applied to a package has no effect. Given that @Target explicitly includes PACKAGE this looks inconsistent to me. A user adding @Deprecated to a package would certainly expect *some* effect, no? JLS argues about the same programmer developing the package and managing its export from then enclosing module. I fail to see how this justifies the decision to let deprecation of packages go without warning. If a library developer deprecates an exported package, shouldn't this signal to consumers of the library, that they should migrate away from using types within that package? Currently it doesn't, because consumers will see no effect of that deprecation. Modules: I could find no reference in JLS specifying any relation between modules and deprecation. In the javadoc of @Deprecated, however, I find some additions (apparently not provided/reviewed via jigsaw), notably: "A module being deprecated does not cause warnings to be issued for uses of types within the module." I assume "within the module" binds to "types" not "uses", right? (nitpick: "within the module" is not a specified term, "associated with" is). For a named module, this makes sense, since "requires" will already raise a warning for the deprecated module, and hence flagging each use of a type from that module adds little value. For unnamed modules, however, this seems to completely bypass any module- level deprecation. There is no "requires" that could be flagged, and use of types from the deprecated modules isn't flagged either. Ergo: most today's consumers of libraries that deprecate either a module or a package will not be informed about this deprecation by compilers. If the rules of JEP 261 define the set of default root modules in a way that ensures that no deprecated modules are in the module graph, then I could imagine as the very minimum that any command line option that pulls in a deprecated module (--add-modules, --modulepath, maybe more), will trigger a warning. The only real problem with that approach is that there is no place where these warnings could be individually suppressed (unless new options were introduced like --add-deprecated-modules ...). I see two consequences from the above: There's little to no incentive to use @Deprecated at any level greater than types, because it is not reliably communicated to users. We keep thinking in terms of types, not modules (not even packages). Second, developers of unnamed modules will see fewer deprecation warnings. Hence, migrating from unnamed to named has an additional burden of having to deal with warnings that could have been dealt with before, but simply weren't visible. Are there any plans to improve in this area? Stephan PS: I also checked if JEP 277 has a say on these issues, but the JEP page doens't give any hints in this direction. As always please let us know if there is more recent information regarding any of the closed JEPs. From rafael.wth at gmail.com Sun Apr 1 21:02:25 2018 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Sun, 1 Apr 2018 23:02:25 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links Message-ID: Hello, I am the/a maintainer of the libraries Byte Buddy, cglib and Mockito with countless dependencies upstream and I wanted to give a summary of adopting the JPMS and migrating away from sun.misc.Unsafe. 1. Java agents cannot define auxiliary classes. Byte Buddy does support the JPMS fully, however, it still relies on sun.misc.Unsafe::defineClass for its Java agent API and currently breaks on Java 11 as this method was removed in a recent EA build. The reason for using Unsafe is that many instrumentations need to define auxiliary classes to aid an instrumentation similar to javac which sometimes needs to define anonymous classes or even synthetic classes. For example, if a Java agent wants to register an event listener to some framework, such listeners often declare multiple methods what makes it impossible to fullfil the listener contract using a lambda expression. Instead, one typically injects an additional class into the same package as the instrumented class. In this case, it is not possible to use MethodHandles.Lookup::defineClass as the class file transformer does not necessarily have private access to the lookup of the instrumented class. The current workarounds are: a) Open the package jdk.internal.misc to gain access to this package's Unsafe class. This can be done via Instrumentation::redefineModule. b) Open the java.lang package to access ClassLoader via reflection. c) Open the java.lang package to access the internal lookup with global access rights. Of these solutions only (b) relies on standard API and is guaranteed to function in the future but the solution still feels hacky and does not work for instrumentations of classes on the bootstrap loader. Opening packages also implies a risk of being applied carelessly since opening the package to the agent's module most likely opens the package to the unnamed module of the system class loader what invites to breaches of the JPMS encapsulation by code that does not ship with the agent. To offer a better solution, I would like to suggest one of the following: a) Add a method defineClass(ClassLoader, byte[], ProtectionDomain) to the Instrumentation interface that works similar to Unsafe::defineClass. This would provide a very simple migration path. Since agents have access to jdk.internal.misc, adding this method does not add any capabilities to the agent, it merley avoids using internal API that might change. b) Supply a MethodHandles.Lookup instance to the ClassFileTransformer::transform API where the instance represents the instrumented class's access rights. This does however carry the risk of invoking the lookupClass method which would either load the instrumented class prematurely causing a circularity error or return an unexpected value such as null. Since the lookup API generally relies on loaded types, there are probably other problems such as invoking Lookup::privateLookupIn before all involved types are loaded. For the sake of simlicity and since easy migration paths make a quick adoption easier, I would suggestion solution (a), also in the light that quick and dirty migrations might still choose option (b) to save time and also since (b) might cause problems when types are not yet loaded. 2. Java proxies cannot invoke default methods of proxied interfaces The Java proxy API does not currently allow the invocation of an overridden default method since the InvocationHandler API only supplies an instance of java.lang.reflection.Method. In Java 8, it was always possible to get hold of method handle of any method of the proxied interface and to specialize the handle on the interface type to invoke the default implementation. With the JPMS, even if an interface type is exported, this same type might not be opened to another module. This implies that if an InvocationHandler is defined by this module to which the interface is exported, this module's InvocationHandler cannot resolve a specialized method handle to a default method of the proxied interface. As a matter of fact, such a call is impossible in this scenario whereas the same call is possible if the proxy is implemented manually at compile time. As a solution, I suggest to provide an argument to the InvocationHandler that represents a lookup instance with the access rights of the proxy class. Using this lookup, the specialized method handles could be resolved. 3. Mocking and serialization libraries still require Unsafe::allocateInstance. For Mockito, it is required to instantiate any class without invoking a constructor with potential side-effects. This is of course a grose violation of Java's general contract for class instantiation but this feature is very useful. Using a Java agent, it is already possible to emulate this feature without internal API by instrumenting all constructors of all classes in the hierarchy of a mocked class by transforming all constructors into the following pseudo-code: SomeConstructor(Void arg) { if (MockitoBootHelper.THREAD_LOCAL.isMockInstantiatingMode()) { super(null); // any constructor of the super class with default values for all arguments } else { // original constructor code... } } Before instantiating a mock, the thread local value that is held by a bootstrap-loader injected class is set to true such that a side-effect free construction is achieved. This is of course too expensive and has negative effects on performance due to additional branching and JIT-byte code limits such that one would rather open jdk.internal.misc to access the Unsafe instantiation mechanism if a Java agent is already available. However, mocking and serialization libraries are not typically loaded as a Java agent. Also, I think that the actual requirements are different. My suggestion here is: a) For serialization libraries, I think that adding MethodHandles.Lookup::newInstance(Class) with standard deserialization mechanics would be sufficient. b) For mocking libraries, this does not suffice as mocks can be of any class. I understand that this breaks encapsulation but for unit tests, I argue that the benefit of using these libraries outweights the benefit of full encapsulation within a unit test. As Mockito is typically run within a build which uses a JDK, we could attach to the current VM using the attachment API. Since Java 9, this is no longer possible to attach to the same VM but instead we start a helper JVM process that applies the attachment indirectly. Unfortunately, this is a rather costly operation what is especially problematic when running a single unit test. (The difference to this approach over Unsafe is about half a second on average.) To overcome this, I would like to suggest to: a) Add a method Instrumentation::allocateInstance(Class). Java agents can already emulate this privilege as described above, this is therefore merely a convenience. b) Add a module jdk.test to JDK 11 with a class JavaTest::getInstrumentation that returns an instrumentation instance for the current JVM. This module should not be resolved by default but only when requiring it explicitly similar to the EE modules after Java 9. I think this solution carries two benefits: a) Test libraries like Mockito can only be used in a testing scope. We experience regularly that Mockito is used in production environments. The library is not meant for that and we warn and document that this is not intended use but people regularly ignore this directive. By requiring this module, this form of abuse would no longer be possible and the JVM could even give a meaningful error message if this use was intended. b) The Instrumentation instance can be used for other meaningful testing approaches. For example, Mockito offers an inline mock maker where the mocking logic is inlined into a method body rather then creating a subclass. This approach mainly targets final classes which have become more common especially since the Kotlin language became popular. To supply this alternative mock maker, Mockito attempts attaching an agent to the current VM (directly or indirectly, depending on the VM's version) which suffers the additional costs for attaching an agent that I mentioned before. Thanks for reading this and I hope that you can consider these, as of today, very practiced use cases. The JPMS migration has gone quite well, I find. With these outstanding problems, JDK 11 could be the first release where a majority of Java programs would no longer need to rely on internal API. Best regards, Rafael From david.lloyd at redhat.com Mon Apr 2 14:58:47 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Mon, 2 Apr 2018 09:58:47 -0500 Subject: Proposed convention for multi-release source JARs Message-ID: Debugging multi-release JARs can be a pain, because their corresponding sources generally do not include the multi-release overlays. I'd like to propose a convention for multi-release source JARs. AFAIK there is no specification for source JARs, but Maven and other build tools commonly output source JARs as a part of their build, hence the term "convention" instead of "standard". The convention would be simple: the source JAR should include META-INF/versions directories exactly as specified in JEP 238. IDEs and other tooling should consider the running or configured project JDK version when determining which sources to examine or display to the user. I don't think the Multi-Release attribute should be required for source JARs. That attribute would be gleaned from the corresponding .class JAR; requiring it in the source JAR seems redundant. WDYT? -- - DML From alex.buckley at oracle.com Tue Apr 3 00:49:46 2018 From: alex.buckley at oracle.com (Alex Buckley) Date: Mon, 02 Apr 2018 17:49:46 -0700 Subject: Deprecation beyond the level of types In-Reply-To: References: Message-ID: <5AC2CFAA.3050704@oracle.com> On 4/1/2018 4:31 AM, Stephan Herrmann wrote: > Packages: JLS & javadoc agree that @Deprecated applied to a package > has no effect. Given that @Target explicitly includes PACKAGE this > looks inconsistent to me. A user adding @Deprecated to a package would > certainly expect *some* effect, no? I understand that JSR 175 forgot to restrict the targets of the Deprecated annotation type, which meant it was possible to write @Deprecated on a package declaration. This should have been disallowed, because (as you imply) no-one conceived of a warning or an error when the package is imported/used. The targets of Deprecated were specified in Java SE 7 as part of the background work for JSR 308; PACKAGE was included for source compatibility with package declarations that already had @Deprecated. > JLS argues about the same programmer developing the package and > managing its export from then enclosing module. I fail to see how this > justifies the decision to let deprecation of packages go without warning. > If a library developer deprecates an exported package, shouldn't this > signal to consumers of the library, that they should migrate away from > using types within that package? Currently it doesn't, because consumers > will see no effect of that deprecation. The JLS "argument" that you mention is a corner case that concerns a deprecated package being exported in a qualified fashion. It's not meant to be a general justification "to let deprecation of packages go without warning". The reason why exporting a deprecated package doesn't generate a warning is because no other use of a deprecated package generates a warning. (And note JEP 211, which reduced the population of warnings for a deprecated package even further, at `import`.) Of course, we could change this behavior, and have long discussions about whether deprecating a package means "deprecate the package, not its types" or "deprecate its types, not the package". But it's not on the radar at this time. > Modules: I could find no reference in JLS specifying any relation between > modules and deprecation. I enumerated "program elements" in the first sentence of JLS 9.6.4.6 specifically to show that "modules" are as subject to deprecation as classes, fields, etc. It's easily concluded from the section that, say, requiring a deprecated module (directly or indirectly) should cause a warning. > In the javadoc of @Deprecated, however, I find > some additions (apparently not provided/reviewed via jigsaw), notably: > "A module being deprecated does not cause warnings to be issued for > uses of types within the module." > For a named module, this makes sense, since "requires" will already raise > a warning for the deprecated module, and hence flagging each use of a type > from that module adds little value. > For unnamed modules, however, this seems to completely bypass any module- > level deprecation. There is no "requires" that could be flagged, and use > of types from the deprecated modules isn't flagged either. > > Ergo: most today's consumers of libraries that deprecate either a module > or a package will not be informed about this deprecation by compilers. ... > I see two consequences from the above: > There's little to no incentive to use @Deprecated at any level greater than > types, because it is not reliably communicated to users. We keep thinking > in terms of types, not modules (not even packages). > Second, developers of unnamed modules will see fewer deprecation warnings. > Hence, migrating from unnamed to named has an additional burden of having > to deal with warnings that could have been dealt with before, but simply > weren't visible. You're right -- the JLS does not mandate a warning when compiling code on the classpath that uses code in deprecated modules. But it's short-sighted to say that deprecation warnings mandated by modularization are a burden; they communicate important information which couldn't be communicated by the compiler before. Alex From henri.tremblay at gmail.com Tue Apr 3 19:26:29 2018 From: henri.tremblay at gmail.com (Henri Tremblay) Date: Tue, 3 Apr 2018 15:26:29 -0400 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: Message-ID: Hi Rafael, I don't like the idea to have to explicitly load a module to create mocks. Because Unsafe.allocateInstance is of course used to create mock but also proxies. For instance, Spring uses it extensively through Objenesis. And different frameworks doing serialization. Also, It means Gradle and Maven would need to be updated to load the module automatically in test context. And all IDE. A lot of annoyance for not much benefit since everybody is currently happy. I've been advocating that if creating a class without calling a constructor is useless in multiple contexts, it should be easily available through the API. I don't know exactly where to put it but it should be - Available to all (java.base) - Protected by the security manager For completeness, there are 4 ways to create a class without calling a constructor right now that I'm aware of: - Unsafe.allocateInstance - sun.reflect.ReflectionFactory.newConstructorForSerialization (my favorite because it's the fastest one) - Generate an extending class forgetting to call the super constructor (so it's not exactly that same class that is instantiated). It requires -Xverify:none - Generate a class extending MagicAccessorImpl that will then instantiates the wanted class but calling the wrong constructor Regards, Henri On 1 April 2018 at 17:02, Rafael Winterhalter wrote: > Hello, > > I am the/a maintainer of the libraries Byte Buddy, cglib and Mockito with > countless dependencies upstream and I wanted to give a summary of adopting > the JPMS and migrating away from sun.misc.Unsafe. > > 1. Java agents cannot define auxiliary classes. > > Byte Buddy does support the JPMS fully, however, it still relies on > sun.misc.Unsafe::defineClass for its Java agent API and currently breaks on > Java 11 as this method was removed in a recent EA build. The reason for > using Unsafe is that many instrumentations need to define auxiliary classes > to aid an instrumentation similar to javac which sometimes needs to define > anonymous classes or even synthetic classes. For example, if a Java agent > wants to register an event listener to some framework, such listeners often > declare multiple methods what makes it impossible to fullfil the listener > contract using a lambda expression. Instead, one typically injects an > additional class into the same package as the instrumented class. In this > case, it is not possible to use MethodHandles.Lookup::defineClass as the > class file transformer does not necessarily have private access to the > lookup of the instrumented class. > > The current workarounds are: > > a) Open the package jdk.internal.misc to gain access to this package's > Unsafe class. This can be done via Instrumentation::redefineModule. > b) Open the java.lang package to access ClassLoader via reflection. > c) Open the java.lang package to access the internal lookup with global > access rights. > > Of these solutions only (b) relies on standard API and is guaranteed to > function in the future but the solution still feels hacky and does not work > for instrumentations of classes on the bootstrap loader. Opening packages > also implies a risk of being applied carelessly since opening the package > to the agent's module most likely opens the package to the unnamed module > of the system class loader what invites to breaches of the JPMS > encapsulation by code that does not ship with the agent. > > To offer a better solution, I would like to suggest one of the following: > > a) Add a method defineClass(ClassLoader, byte[], ProtectionDomain) to the > Instrumentation interface that works similar to Unsafe::defineClass. This > would provide a very simple migration path. Since agents have access to > jdk.internal.misc, adding this method does not add any capabilities to the > agent, it merley avoids using internal API that might change. > b) Supply a MethodHandles.Lookup instance to the > ClassFileTransformer::transform API where the instance represents the > instrumented class's access rights. This does however carry the risk of > invoking the lookupClass method which would either load the instrumented > class prematurely causing a circularity error or return an unexpected value > such as null. Since the lookup API generally relies on loaded types, there > are probably other problems such as invoking Lookup::privateLookupIn before > all involved types are loaded. > > For the sake of simlicity and since easy migration paths make a quick > adoption easier, I would suggestion solution (a), also in the light that > quick and dirty migrations might still choose option (b) to save time and > also since (b) might cause problems when types are not yet loaded. > > 2. Java proxies cannot invoke default methods of proxied interfaces > > The Java proxy API does not currently allow the invocation of an overridden > default method since > the InvocationHandler API only supplies an instance of > java.lang.reflection.Method. In Java 8, it was always possible to get hold > of method handle of any method of the proxied interface and to specialize > the handle on the interface type to invoke the default implementation. With > the JPMS, even if an interface type is exported, this same type might not > be opened to another module. This implies that if an InvocationHandler is > defined by this module to which the interface is exported, this module's > InvocationHandler cannot resolve a specialized method handle to a default > method of the proxied interface. As a matter of fact, such a call is > impossible in this scenario whereas the same call is possible if the proxy > is implemented manually at compile time. > > As a solution, I suggest to provide an argument to the InvocationHandler > that represents a lookup instance with the access rights of the proxy > class. Using this lookup, the specialized method handles could be resolved. > > 3. Mocking and serialization libraries still require > Unsafe::allocateInstance. > > For Mockito, it is required to instantiate any class without invoking a > constructor with potential side-effects. This is of course a grose > violation of Java's general contract for class instantiation but this > feature is very useful. > > Using a Java agent, it is already possible to emulate this feature without > internal API by instrumenting all constructors of all classes in the > hierarchy of a mocked class by transforming all constructors into the > following pseudo-code: > > SomeConstructor(Void arg) { > if (MockitoBootHelper.THREAD_LOCAL.isMockInstantiatingMode()) { > super(null); // any constructor of the super class with default values > for all arguments > } else { > // original constructor code... > } > } > > Before instantiating a mock, the thread local value that is held by a > bootstrap-loader injected class is set to true such that a side-effect free > construction is achieved. > > This is of course too expensive and has negative effects on performance due > to additional branching and JIT-byte code limits such that one would rather > open jdk.internal.misc to access the Unsafe instantiation mechanism if a > Java agent is already available. > > However, mocking and serialization libraries are not typically loaded as a > Java agent. Also, I think that the actual requirements are different. My > suggestion here is: > > a) For serialization libraries, I think that adding > MethodHandles.Lookup::newInstance(Class) with > standard deserialization mechanics would be sufficient. > b) For mocking libraries, this does not suffice as mocks can be of any > class. I understand that this breaks encapsulation but for unit tests, I > argue that the benefit of using these libraries outweights the benefit of > full encapsulation within a unit test. > > As Mockito is typically run within a build which uses a JDK, we could > attach to the current VM using the attachment API. Since Java 9, this is no > longer possible to attach to the same VM but instead we start a helper JVM > process that applies the attachment indirectly. Unfortunately, this is a > rather costly operation what is especially problematic when running a > single unit test. (The difference to this approach over Unsafe is about > half a second on average.) > > To overcome this, I would like to suggest to: > > a) Add a method Instrumentation::allocateInstance(Class). Java agents can > already emulate this privilege as described above, this is therefore merely > a convenience. > b) Add a module jdk.test to JDK 11 with a class > JavaTest::getInstrumentation that returns an instrumentation instance for > the current JVM. This module should not be resolved by default but only > when requiring it explicitly similar to the EE modules after Java 9. > > I think this solution carries two benefits: > > a) Test libraries like Mockito can only be used in a testing scope. We > experience regularly that Mockito is used in production environments. The > library is not meant for that and we warn and document that this is not > intended use but people regularly ignore this directive. By requiring this > module, this form of abuse would no longer be possible and the JVM could > even give a meaningful error message if this use was intended. > b) The Instrumentation instance can be used for other meaningful testing > approaches. For example, Mockito offers an inline mock maker where the > mocking logic is inlined into a method body rather then creating a > subclass. This approach mainly targets final classes which have become more > common especially since the Kotlin language became popular. To supply this > alternative mock maker, Mockito attempts attaching an agent to the current > VM (directly or indirectly, depending on the VM's version) which suffers > the additional costs for attaching an agent that I mentioned before. > > Thanks for reading this and I hope that you can consider these, as of > today, very practiced use cases. The JPMS migration has gone quite well, I > find. With these outstanding problems, JDK 11 could be the first release > where a majority of Java programs would no longer need to rely on internal > API. > > Best regards, Rafael > From russell.gold at oracle.com Tue Apr 3 19:48:41 2018 From: russell.gold at oracle.com (Russell Gold) Date: Tue, 3 Apr 2018 15:48:41 -0400 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: Message-ID: Hi Rafael, This is the point of multi-release jars. There is a new supported way of creating classes. As of Java 9, you are supposed to do: > return MethodHandles.privateLookupIn(anchorClass, MethodHandles.lookup()) > .dropLookupMode(MethodHandles.Lookup.PRIVATE); > .defineClass(classBytes); which doesn?t work before then, but does work in JDK 11. It requires some rework of your code, unfortunately, since you now have to pass in an anchor class to define the package in which your new class will live. I?ve just gotten this working in SimpleStub , which also uses my easier way to create MR Jars. - Russ > On Apr 1, 2018, at 5:02 PM, Rafael Winterhalter wrote: > > Hello, > > I am the/a maintainer of the libraries Byte Buddy, cglib and Mockito with > countless dependencies upstream and I wanted to give a summary of adopting > the JPMS and migrating away from sun.misc.Unsafe. > > 1. Java agents cannot define auxiliary classes. > > Byte Buddy does support the JPMS fully, however, it still relies on > sun.misc.Unsafe::defineClass for its Java agent API and currently breaks on > Java 11 as this method was removed in a recent EA build. The reason for > using Unsafe is that many instrumentations need to define auxiliary classes > to aid an instrumentation similar to javac which sometimes needs to define > anonymous classes or even synthetic classes. For example, if a Java agent > wants to register an event listener to some framework, such listeners often > declare multiple methods what makes it impossible to fullfil the listener > contract using a lambda expression. Instead, one typically injects an > additional class into the same package as the instrumented class. In this > case, it is not possible to use MethodHandles.Lookup::defineClass as the > class file transformer does not necessarily have private access to the > lookup of the instrumented class. > > The current workarounds are: > > a) Open the package jdk.internal.misc to gain access to this package's > Unsafe class. This can be done via Instrumentation::redefineModule. > b) Open the java.lang package to access ClassLoader via reflection. > c) Open the java.lang package to access the internal lookup with global > access rights. > > Of these solutions only (b) relies on standard API and is guaranteed to > function in the future but the solution still feels hacky and does not work > for instrumentations of classes on the bootstrap loader. Opening packages > also implies a risk of being applied carelessly since opening the package > to the agent's module most likely opens the package to the unnamed module > of the system class loader what invites to breaches of the JPMS > encapsulation by code that does not ship with the agent. > > To offer a better solution, I would like to suggest one of the following: > > a) Add a method defineClass(ClassLoader, byte[], ProtectionDomain) to the > Instrumentation interface that works similar to Unsafe::defineClass. This > would provide a very simple migration path. Since agents have access to > jdk.internal.misc, adding this method does not add any capabilities to the > agent, it merley avoids using internal API that might change. > b) Supply a MethodHandles.Lookup instance to the > ClassFileTransformer::transform API where the instance represents the > instrumented class's access rights. This does however carry the risk of > invoking the lookupClass method which would either load the instrumented > class prematurely causing a circularity error or return an unexpected value > such as null. Since the lookup API generally relies on loaded types, there > are probably other problems such as invoking Lookup::privateLookupIn before > all involved types are loaded. > > For the sake of simlicity and since easy migration paths make a quick > adoption easier, I would suggestion solution (a), also in the light that > quick and dirty migrations might still choose option (b) to save time and > also since (b) might cause problems when types are not yet loaded. > > 2. Java proxies cannot invoke default methods of proxied interfaces > > The Java proxy API does not currently allow the invocation of an overridden > default method since > the InvocationHandler API only supplies an instance of > java.lang.reflection.Method. In Java 8, it was always possible to get hold > of method handle of any method of the proxied interface and to specialize > the handle on the interface type to invoke the default implementation. With > the JPMS, even if an interface type is exported, this same type might not > be opened to another module. This implies that if an InvocationHandler is > defined by this module to which the interface is exported, this module's > InvocationHandler cannot resolve a specialized method handle to a default > method of the proxied interface. As a matter of fact, such a call is > impossible in this scenario whereas the same call is possible if the proxy > is implemented manually at compile time. > > As a solution, I suggest to provide an argument to the InvocationHandler > that represents a lookup instance with the access rights of the proxy > class. Using this lookup, the specialized method handles could be resolved. > > 3. Mocking and serialization libraries still require > Unsafe::allocateInstance. > > For Mockito, it is required to instantiate any class without invoking a > constructor with potential side-effects. This is of course a grose > violation of Java's general contract for class instantiation but this > feature is very useful. > > Using a Java agent, it is already possible to emulate this feature without > internal API by instrumenting all constructors of all classes in the > hierarchy of a mocked class by transforming all constructors into the > following pseudo-code: > > SomeConstructor(Void arg) { > if (MockitoBootHelper.THREAD_LOCAL.isMockInstantiatingMode()) { > super(null); // any constructor of the super class with default values > for all arguments > } else { > // original constructor code... > } > } > > Before instantiating a mock, the thread local value that is held by a > bootstrap-loader injected class is set to true such that a side-effect free > construction is achieved. > > This is of course too expensive and has negative effects on performance due > to additional branching and JIT-byte code limits such that one would rather > open jdk.internal.misc to access the Unsafe instantiation mechanism if a > Java agent is already available. > > However, mocking and serialization libraries are not typically loaded as a > Java agent. Also, I think that the actual requirements are different. My > suggestion here is: > > a) For serialization libraries, I think that adding > MethodHandles.Lookup::newInstance(Class) with > standard deserialization mechanics would be sufficient. > b) For mocking libraries, this does not suffice as mocks can be of any > class. I understand that this breaks encapsulation but for unit tests, I > argue that the benefit of using these libraries outweights the benefit of > full encapsulation within a unit test. > > As Mockito is typically run within a build which uses a JDK, we could > attach to the current VM using the attachment API. Since Java 9, this is no > longer possible to attach to the same VM but instead we start a helper JVM > process that applies the attachment indirectly. Unfortunately, this is a > rather costly operation what is especially problematic when running a > single unit test. (The difference to this approach over Unsafe is about > half a second on average.) > > To overcome this, I would like to suggest to: > > a) Add a method Instrumentation::allocateInstance(Class). Java agents can > already emulate this privilege as described above, this is therefore merely > a convenience. > b) Add a module jdk.test to JDK 11 with a class > JavaTest::getInstrumentation that returns an instrumentation instance for > the current JVM. This module should not be resolved by default but only > when requiring it explicitly similar to the EE modules after Java 9. > > I think this solution carries two benefits: > > a) Test libraries like Mockito can only be used in a testing scope. We > experience regularly that Mockito is used in production environments. The > library is not meant for that and we warn and document that this is not > intended use but people regularly ignore this directive. By requiring this > module, this form of abuse would no longer be possible and the JVM could > even give a meaningful error message if this use was intended. > b) The Instrumentation instance can be used for other meaningful testing > approaches. For example, Mockito offers an inline mock maker where the > mocking logic is inlined into a method body rather then creating a > subclass. This approach mainly targets final classes which have become more > common especially since the Kotlin language became popular. To supply this > alternative mock maker, Mockito attempts attaching an agent to the current > VM (directly or indirectly, depending on the VM's version) which suffers > the additional costs for attaching an agent that I mentioned before. > > Thanks for reading this and I hope that you can consider these, as of > today, very practiced use cases. The JPMS migration has gone quite well, I > find. With these outstanding problems, JDK 11 could be the first release > where a majority of Java programs would no longer need to rely on internal > API. > > Best regards, Rafael From russell.gold at oracle.com Tue Apr 3 19:49:34 2018 From: russell.gold at oracle.com (Russell Gold) Date: Tue, 3 Apr 2018 15:49:34 -0400 Subject: Easy multirelease jars Message-ID: <7C24C46C-A9D9-497A-9149-54B508CA31A7@oracle.com> have just developed a new and easier way for building MR Jars , while waiting for the capability to be built into Maven. This approach is not only simple to set up (just use the designated parent POM, if you can), it lets you unit test for any supported JDK. For example: jdk7 && mvn -Dmulti_release clean test jdk10 && mvn -Dmulti_release clean test where jdk7 and jdk10 set the appropriate versions for maven to use. Either will run against the appropriate additional code. To build an MR JAR you set a property on the command line mvn -Dmulti_release clean install which happens automatically when doing a release. I have also explained how it works at Easier Than It Looks From russell.gold at oracle.com Tue Apr 3 20:05:39 2018 From: russell.gold at oracle.com (Russell Gold) Date: Tue, 3 Apr 2018 16:05:39 -0400 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: Message-ID: Meh. Answered without thinking it through. I love this basic approach: >>> b) Add a module jdk.test to JDK 11 with a class >>> JavaTest::getInstrumentation that returns an instrumentation instance for >>> the current JVM. This module should not be resolved by default but only >>> when requiring it explicitly similar to the EE modules after Java 9. Unit testing is a special case, and it seems more than reasonable to me that you should be able to do things when running unit tests that would never be permitted in a production environment. Since this module would only be resolved explicitly, it could contain capabilities customized for this special case, including things that currently use Unsafe or involve what appear to be hacks in the JDK. That said, I do think that it is still possible to make a lot of unit testing libraries work, but it is getting harder. > On Apr 3, 2018, at 3:48 PM, Russell Gold wrote: > > Hi Rafael, > > This is the point of multi-release jars. There is a new supported way of creating classes. As of Java 9, you are supposed to do: > >> return MethodHandles.privateLookupIn(anchorClass, MethodHandles.lookup()) >> .dropLookupMode(MethodHandles.Lookup.PRIVATE); >> .defineClass(classBytes); > > which doesn?t work before then, but does work in JDK 11. It requires some rework of your code, unfortunately, since you now have to pass in an anchor class to define the package in which your new class will live. I?ve just gotten this working in SimpleStub , which also uses my easier way to create MR Jars. > > - Russ > > >> On Apr 1, 2018, at 5:02 PM, Rafael Winterhalter wrote: >> >> Hello, >> >> I am the/a maintainer of the libraries Byte Buddy, cglib and Mockito with >> countless dependencies upstream and I wanted to give a summary of adopting >> the JPMS and migrating away from sun.misc.Unsafe. >> >> 1. Java agents cannot define auxiliary classes. >> >> Byte Buddy does support the JPMS fully, however, it still relies on >> sun.misc.Unsafe::defineClass for its Java agent API and currently breaks on >> Java 11 as this method was removed in a recent EA build. The reason for >> using Unsafe is that many instrumentations need to define auxiliary classes >> to aid an instrumentation similar to javac which sometimes needs to define >> anonymous classes or even synthetic classes. For example, if a Java agent >> wants to register an event listener to some framework, such listeners often >> declare multiple methods what makes it impossible to fullfil the listener >> contract using a lambda expression. Instead, one typically injects an >> additional class into the same package as the instrumented class. In this >> case, it is not possible to use MethodHandles.Lookup::defineClass as the >> class file transformer does not necessarily have private access to the >> lookup of the instrumented class. >> >> The current workarounds are: >> >> a) Open the package jdk.internal.misc to gain access to this package's >> Unsafe class. This can be done via Instrumentation::redefineModule. >> b) Open the java.lang package to access ClassLoader via reflection. >> c) Open the java.lang package to access the internal lookup with global >> access rights. >> >> Of these solutions only (b) relies on standard API and is guaranteed to >> function in the future but the solution still feels hacky and does not work >> for instrumentations of classes on the bootstrap loader. Opening packages >> also implies a risk of being applied carelessly since opening the package >> to the agent's module most likely opens the package to the unnamed module >> of the system class loader what invites to breaches of the JPMS >> encapsulation by code that does not ship with the agent. >> >> To offer a better solution, I would like to suggest one of the following: >> >> a) Add a method defineClass(ClassLoader, byte[], ProtectionDomain) to the >> Instrumentation interface that works similar to Unsafe::defineClass. This >> would provide a very simple migration path. Since agents have access to >> jdk.internal.misc, adding this method does not add any capabilities to the >> agent, it merley avoids using internal API that might change. >> b) Supply a MethodHandles.Lookup instance to the >> ClassFileTransformer::transform API where the instance represents the >> instrumented class's access rights. This does however carry the risk of >> invoking the lookupClass method which would either load the instrumented >> class prematurely causing a circularity error or return an unexpected value >> such as null. Since the lookup API generally relies on loaded types, there >> are probably other problems such as invoking Lookup::privateLookupIn before >> all involved types are loaded. >> >> For the sake of simlicity and since easy migration paths make a quick >> adoption easier, I would suggestion solution (a), also in the light that >> quick and dirty migrations might still choose option (b) to save time and >> also since (b) might cause problems when types are not yet loaded. >> >> 2. Java proxies cannot invoke default methods of proxied interfaces >> >> The Java proxy API does not currently allow the invocation of an overridden >> default method since >> the InvocationHandler API only supplies an instance of >> java.lang.reflection.Method. In Java 8, it was always possible to get hold >> of method handle of any method of the proxied interface and to specialize >> the handle on the interface type to invoke the default implementation. With >> the JPMS, even if an interface type is exported, this same type might not >> be opened to another module. This implies that if an InvocationHandler is >> defined by this module to which the interface is exported, this module's >> InvocationHandler cannot resolve a specialized method handle to a default >> method of the proxied interface. As a matter of fact, such a call is >> impossible in this scenario whereas the same call is possible if the proxy >> is implemented manually at compile time. >> >> As a solution, I suggest to provide an argument to the InvocationHandler >> that represents a lookup instance with the access rights of the proxy >> class. Using this lookup, the specialized method handles could be resolved. >> >> 3. Mocking and serialization libraries still require >> Unsafe::allocateInstance. >> >> For Mockito, it is required to instantiate any class without invoking a >> constructor with potential side-effects. This is of course a grose >> violation of Java's general contract for class instantiation but this >> feature is very useful. >> >> Using a Java agent, it is already possible to emulate this feature without >> internal API by instrumenting all constructors of all classes in the >> hierarchy of a mocked class by transforming all constructors into the >> following pseudo-code: >> >> SomeConstructor(Void arg) { >> if (MockitoBootHelper.THREAD_LOCAL.isMockInstantiatingMode()) { >> super(null); // any constructor of the super class with default values >> for all arguments >> } else { >> // original constructor code... >> } >> } >> >> Before instantiating a mock, the thread local value that is held by a >> bootstrap-loader injected class is set to true such that a side-effect free >> construction is achieved. >> >> This is of course too expensive and has negative effects on performance due >> to additional branching and JIT-byte code limits such that one would rather >> open jdk.internal.misc to access the Unsafe instantiation mechanism if a >> Java agent is already available. >> >> However, mocking and serialization libraries are not typically loaded as a >> Java agent. Also, I think that the actual requirements are different. My >> suggestion here is: >> >> a) For serialization libraries, I think that adding >> MethodHandles.Lookup::newInstance(Class) with >> standard deserialization mechanics would be sufficient. >> b) For mocking libraries, this does not suffice as mocks can be of any >> class. I understand that this breaks encapsulation but for unit tests, I >> argue that the benefit of using these libraries outweights the benefit of >> full encapsulation within a unit test. >> >> As Mockito is typically run within a build which uses a JDK, we could >> attach to the current VM using the attachment API. Since Java 9, this is no >> longer possible to attach to the same VM but instead we start a helper JVM >> process that applies the attachment indirectly. Unfortunately, this is a >> rather costly operation what is especially problematic when running a >> single unit test. (The difference to this approach over Unsafe is about >> half a second on average.) >> >> To overcome this, I would like to suggest to: >> >> a) Add a method Instrumentation::allocateInstance(Class). Java agents can >> already emulate this privilege as described above, this is therefore merely >> a convenience. >> b) Add a module jdk.test to JDK 11 with a class >> JavaTest::getInstrumentation that returns an instrumentation instance for >> the current JVM. This module should not be resolved by default but only >> when requiring it explicitly similar to the EE modules after Java 9. >> >> I think this solution carries two benefits: >> >> a) Test libraries like Mockito can only be used in a testing scope. We >> experience regularly that Mockito is used in production environments. The >> library is not meant for that and we warn and document that this is not >> intended use but people regularly ignore this directive. By requiring this >> module, this form of abuse would no longer be possible and the JVM could >> even give a meaningful error message if this use was intended. >> b) The Instrumentation instance can be used for other meaningful testing >> approaches. For example, Mockito offers an inline mock maker where the >> mocking logic is inlined into a method body rather then creating a >> subclass. This approach mainly targets final classes which have become more >> common especially since the Kotlin language became popular. To supply this >> alternative mock maker, Mockito attempts attaching an agent to the current >> VM (directly or indirectly, depending on the VM's version) which suffers >> the additional costs for attaching an agent that I mentioned before. >> >> Thanks for reading this and I hope that you can consider these, as of >> today, very practiced use cases. The JPMS migration has gone quite well, I >> find. With these outstanding problems, JDK 11 could be the first release >> where a majority of Java programs would no longer need to rely on internal >> API. >> >> Best regards, Rafael > From blackdrag at gmx.org Tue Apr 3 22:24:19 2018 From: blackdrag at gmx.org (Jochen Theodorou) Date: Wed, 4 Apr 2018 00:24:19 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: Message-ID: On 03.04.2018 21:26, Henri Tremblay wrote: [...] > For completeness, there are 4 ways to create a class without calling a > constructor right now that I'm aware of: > > - Unsafe.allocateInstance which is supposed to go away at some point > - sun.reflect.ReflectionFactory.newConstructorForSerialization (my > favorite because it's the fastest one) which afaik works in java9 but is also one of those critical doomed APIs > - Generate an extending class forgetting to call the super constructor > (so it's not exactly that same class that is instantiated). It requires > -Xverify:none Is this really an option for a production environment? > - Generate a class extending MagicAccessorImpl that will then > instantiates the wanted class but calling the wrong constructor Is jdk.internal.reflect.MagicAccessorImpl still usable in Java9+? I thought this is no longer exported Under the premise that all critical API usages will be removed in the future and replacement APIs will be created I think we might indeed still miss something here bye Jochen From rafael.wth at gmail.com Wed Apr 4 12:05:06 2018 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Wed, 4 Apr 2018 14:05:06 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: Message-ID: Hei Henri, for Maven, Gradle and IDEs I think it is sensible to expect that these runners would resolve such a test module unless it is explicitly specified otherwise. Already today, those runners patch the module under test in order to include the test code. I would expect of these runners to adopt similarly to a test module as they adopted to the module system in general. As for Spring's use case, I believe that the approach was chosen such that class proxies resemble interface proxies as much as possible. In general however, I think that Spring could just generate a subclass proxy and delegate to the overridden methods instead of delegating to an instance of the proxied class. If the proxy class mimics the proxied class's constructors' signatures, this proxy can be instantiated exactly like the proxied class would have been. This way, using Objenesis and the like is no longer necessary and I argue that one should avoid exposing this functionality if there is a reasonable alternative that does not require any derivation from Javas programming model. As a matter of fact, I often find developers confused about how fields are not set on such Spring proxies and I even found possible exploits in code because of it. For example, imagine a class: class Foo { public final production; Foo(boolean production) { this.production = production; } // Only call from unit test public Foo() { this(true); } public void doSomethingSensible() { if (production) { doSecurityCheck(); } // do something ... } } With Objenesis, you can create an instance of the above class and set it to test mode with the production field being false as the default value. In contrast, for the case of serialization, I already argued that support for serializable types should probably go into MethodHandles.Lookup similar to how defineClass was added. This does not render an additional security threat, if Foo was serializable I could just synthetically create a serialized vale of an instance with production being false and deserialize this value in a production environment. On a side note, Spring would create a class with the above field value automatically if Foo was a bean, making this exploit very easy even when using a security manager as the proxy instance is normally available to all sorts of code. Finally, I disagree with the argument that this sort of API should be accessible to "standard code" only shielded by a security manager because this feature is useful today. The module system becomes useless the moment you punch even a little hole into it. Therefore, I find it important to remove sun.misc.Unsafe as quickly as possible as the module system can be easily circumvented until it is removed and since the JPMS only show its full benefit once this is no longer possible. But in order to keep testing frameworks that have a good reason to break module boundaries up and running, I think adding a test modules is a reasonable compromise. As an additional benefit, it would allow libraries like Mockito that should not be used in production to self-identify. One could even argue that this ability to instantiate instance is nothing that needs to be exposed to a Java agent so maybe this functionality could also be part of the mentioned test module instead of being a part of instrumentation. On the other hand, if there is a use case not being considered for a Java agent, this agent might be tempted to open jdk.internal.misc to gain this access, one might therefore just decide to add the method to Instrumentation to offer an easy migration path to applications that aim to migrate their code with a minimal delta. Best regards, Rafael 2018-04-03 21:26 GMT+02:00 Henri Tremblay : > Hi Rafael, > > I don't like the idea to have to explicitly load a module to create mocks. > > Because Unsafe.allocateInstance is of course used to create mock but also > proxies. For instance, Spring uses it extensively through Objenesis. And > different frameworks doing serialization. > > Also, It means Gradle and Maven would need to be updated to load the > module automatically in test context. And all IDE. A lot of annoyance for > not much benefit since everybody is currently happy. > > I've been advocating that if creating a class without calling a > constructor is useless in multiple contexts, it should be easily available > through the API. I don't know exactly where to put it but it should be > > - Available to all (java.base) > - Protected by the security manager > > For completeness, there are 4 ways to create a class without calling a > constructor right now that I'm aware of: > > - Unsafe.allocateInstance > - sun.reflect.ReflectionFactory.newConstructorForSerialization (my > favorite because it's the fastest one) > - Generate an extending class forgetting to call the super constructor > (so it's not exactly that same class that is instantiated). It requires > -Xverify:none > - Generate a class extending MagicAccessorImpl that will then > instantiates the wanted class but calling the wrong constructor > > Regards, > Henri > > > > On 1 April 2018 at 17:02, Rafael Winterhalter > wrote: > >> Hello, >> >> I am the/a maintainer of the libraries Byte Buddy, cglib and Mockito with >> countless dependencies upstream and I wanted to give a summary of adopting >> the JPMS and migrating away from sun.misc.Unsafe. >> >> 1. Java agents cannot define auxiliary classes. >> >> Byte Buddy does support the JPMS fully, however, it still relies on >> sun.misc.Unsafe::defineClass for its Java agent API and currently breaks >> on >> Java 11 as this method was removed in a recent EA build. The reason for >> using Unsafe is that many instrumentations need to define auxiliary >> classes >> to aid an instrumentation similar to javac which sometimes needs to define >> anonymous classes or even synthetic classes. For example, if a Java agent >> wants to register an event listener to some framework, such listeners >> often >> declare multiple methods what makes it impossible to fullfil the listener >> contract using a lambda expression. Instead, one typically injects an >> additional class into the same package as the instrumented class. In this >> case, it is not possible to use MethodHandles.Lookup::defineClass as the >> class file transformer does not necessarily have private access to the >> lookup of the instrumented class. >> >> The current workarounds are: >> >> a) Open the package jdk.internal.misc to gain access to this package's >> Unsafe class. This can be done via Instrumentation::redefineModule. >> b) Open the java.lang package to access ClassLoader via reflection. >> c) Open the java.lang package to access the internal lookup with global >> access rights. >> >> Of these solutions only (b) relies on standard API and is guaranteed to >> function in the future but the solution still feels hacky and does not >> work >> for instrumentations of classes on the bootstrap loader. Opening packages >> also implies a risk of being applied carelessly since opening the package >> to the agent's module most likely opens the package to the unnamed module >> of the system class loader what invites to breaches of the JPMS >> encapsulation by code that does not ship with the agent. >> >> To offer a better solution, I would like to suggest one of the following: >> >> a) Add a method defineClass(ClassLoader, byte[], ProtectionDomain) to the >> Instrumentation interface that works similar to Unsafe::defineClass. This >> would provide a very simple migration path. Since agents have access to >> jdk.internal.misc, adding this method does not add any capabilities to the >> agent, it merley avoids using internal API that might change. >> b) Supply a MethodHandles.Lookup instance to the >> ClassFileTransformer::transform API where the instance represents the >> instrumented class's access rights. This does however carry the risk of >> invoking the lookupClass method which would either load the instrumented >> class prematurely causing a circularity error or return an unexpected >> value >> such as null. Since the lookup API generally relies on loaded types, there >> are probably other problems such as invoking Lookup::privateLookupIn >> before >> all involved types are loaded. >> >> For the sake of simlicity and since easy migration paths make a quick >> adoption easier, I would suggestion solution (a), also in the light that >> quick and dirty migrations might still choose option (b) to save time and >> also since (b) might cause problems when types are not yet loaded. >> >> 2. Java proxies cannot invoke default methods of proxied interfaces >> >> The Java proxy API does not currently allow the invocation of an >> overridden >> default method since >> the InvocationHandler API only supplies an instance of >> java.lang.reflection.Method. In Java 8, it was always possible to get hold >> of method handle of any method of the proxied interface and to specialize >> the handle on the interface type to invoke the default implementation. >> With >> the JPMS, even if an interface type is exported, this same type might not >> be opened to another module. This implies that if an InvocationHandler is >> defined by this module to which the interface is exported, this module's >> InvocationHandler cannot resolve a specialized method handle to a default >> method of the proxied interface. As a matter of fact, such a call is >> impossible in this scenario whereas the same call is possible if the proxy >> is implemented manually at compile time. >> >> As a solution, I suggest to provide an argument to the InvocationHandler >> that represents a lookup instance with the access rights of the proxy >> class. Using this lookup, the specialized method handles could be >> resolved. >> >> 3. Mocking and serialization libraries still require >> Unsafe::allocateInstance. >> >> For Mockito, it is required to instantiate any class without invoking a >> constructor with potential side-effects. This is of course a grose >> violation of Java's general contract for class instantiation but this >> feature is very useful. >> >> Using a Java agent, it is already possible to emulate this feature without >> internal API by instrumenting all constructors of all classes in the >> hierarchy of a mocked class by transforming all constructors into the >> following pseudo-code: >> >> SomeConstructor(Void arg) { >> if (MockitoBootHelper.THREAD_LOCAL.isMockInstantiatingMode()) { >> super(null); // any constructor of the super class with default values >> for all arguments >> } else { >> // original constructor code... >> } >> } >> >> Before instantiating a mock, the thread local value that is held by a >> bootstrap-loader injected class is set to true such that a side-effect >> free >> construction is achieved. >> >> This is of course too expensive and has negative effects on performance >> due >> to additional branching and JIT-byte code limits such that one would >> rather >> open jdk.internal.misc to access the Unsafe instantiation mechanism if a >> Java agent is already available. >> >> However, mocking and serialization libraries are not typically loaded as a >> Java agent. Also, I think that the actual requirements are different. My >> suggestion here is: >> >> a) For serialization libraries, I think that adding >> MethodHandles.Lookup::newInstance(Class) with >> standard deserialization mechanics would be sufficient. >> b) For mocking libraries, this does not suffice as mocks can be of any >> class. I understand that this breaks encapsulation but for unit tests, I >> argue that the benefit of using these libraries outweights the benefit of >> full encapsulation within a unit test. >> >> As Mockito is typically run within a build which uses a JDK, we could >> attach to the current VM using the attachment API. Since Java 9, this is >> no >> longer possible to attach to the same VM but instead we start a helper JVM >> process that applies the attachment indirectly. Unfortunately, this is a >> rather costly operation what is especially problematic when running a >> single unit test. (The difference to this approach over Unsafe is about >> half a second on average.) >> >> To overcome this, I would like to suggest to: >> >> a) Add a method Instrumentation::allocateInstance(Class). Java agents can >> already emulate this privilege as described above, this is therefore >> merely >> a convenience. >> b) Add a module jdk.test to JDK 11 with a class >> JavaTest::getInstrumentation that returns an instrumentation instance for >> the current JVM. This module should not be resolved by default but only >> when requiring it explicitly similar to the EE modules after Java 9. >> >> I think this solution carries two benefits: >> >> a) Test libraries like Mockito can only be used in a testing scope. We >> experience regularly that Mockito is used in production environments. The >> library is not meant for that and we warn and document that this is not >> intended use but people regularly ignore this directive. By requiring this >> module, this form of abuse would no longer be possible and the JVM could >> even give a meaningful error message if this use was intended. >> b) The Instrumentation instance can be used for other meaningful testing >> approaches. For example, Mockito offers an inline mock maker where the >> mocking logic is inlined into a method body rather then creating a >> subclass. This approach mainly targets final classes which have become >> more >> common especially since the Kotlin language became popular. To supply this >> alternative mock maker, Mockito attempts attaching an agent to the current >> VM (directly or indirectly, depending on the VM's version) which suffers >> the additional costs for attaching an agent that I mentioned before. >> >> Thanks for reading this and I hope that you can consider these, as of >> today, very practiced use cases. The JPMS migration has gone quite well, I >> find. With these outstanding problems, JDK 11 could be the first release >> where a majority of Java programs would no longer need to rely on internal >> API. >> >> Best regards, Rafael >> > > From Michael.Rasmussen at roguewave.com Wed Apr 4 13:27:18 2018 From: Michael.Rasmussen at roguewave.com (Michael Rasmussen) Date: Wed, 4 Apr 2018 13:27:18 +0000 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: , Message-ID: I can echo a lot of what Rafael said. JRebel is also a heavy user of the mentioned APIs here, especially Unsafe.defineClass and Unsafe.allocateInstance -- we actually use all 4 ways mentioned for creating an instance without calling the constructor (depending on Java version and vendor). Like testing frameworks, JRebel is also something minded for a development environment, and not a production environment. Yes, JRebel can be quite invasive with how we instrument things, and if no official API exists for doing these things, we generally dig in the private APIs until we find it; but an official supported API would always be preferable and to me the Instrumentation API does seems like a good fit for this kind of API. One of the main things about Unsafe.defineClass is that it can define classes in the boot (null) classloader (and in any package). We use this since we sometimes need to define companion classes in the same package as an existing on, including classes defined in the boot classloader. /Michael From: jigsaw-dev on behalf of Jochen Theodorou Sent: 04 April 2018 01:24 To: jigsaw-dev at openjdk.java.net Subject: Re: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links ? On 03.04.2018 21:26, Henri Tremblay wrote: [...] > For completeness, there are 4 ways to create a class without calling a > constructor right now that I'm aware of: > >???? - Unsafe.allocateInstance which is supposed to go away at some point >???? - sun.reflect.ReflectionFactory.newConstructorForSerialization (my >???? favorite because it's the fastest one) which afaik works in java9 but is also one of those critical doomed APIs >???? - Generate an extending class forgetting to call the super constructor >???? (so it's not exactly that same class that is instantiated). It requires >???? -Xverify:none Is this really an option for a production environment? >???? - Generate a class extending MagicAccessorImpl that will then >???? instantiates the wanted class but calling the wrong constructor Is jdk.internal.reflect.MagicAccessorImpl still usable in Java9+? I thought this is no longer exported Under the premise that all critical API usages will be removed in the future and replacement APIs will be created I think we might indeed still miss something here bye Jochen From henri.tremblay at gmail.com Fri Apr 6 03:37:18 2018 From: henri.tremblay at gmail.com (Henri Tremblay) Date: Thu, 5 Apr 2018 23:37:18 -0400 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: Message-ID: Answers inline On 4 April 2018 at 08:05, Rafael Winterhalter wrote: > Hei Henri, > > for Maven, Gradle and IDEs I think it is sensible to expect that these > runners would resolve such a test module unless it is explicitly specified > otherwise. Already today, those runners patch the module under test in > order to include the test code. I would expect of these runners to adopt > similarly to a test module as they adopted to the module system in general. > > -> Yes. I agree. > As for Spring's use case, I believe that the approach was chosen such that > class proxies resemble interface proxies as much as possible. In general > however, I think that Spring could just generate a subclass proxy and > delegate to the overridden methods instead of delegating to an instance of > the proxied class. If the proxy class mimics the proxied class's > constructors' signatures, this proxy can be instantiated exactly like the > proxied class would have been. This way, using Objenesis and the like is no > longer necessary and I argue that one should avoid exposing this > functionality if there is a reasonable alternative that does not require > any derivation from Javas programming model. As a matter of fact, I often > find developers confused about how fields are not set on such Spring > proxies and I even found possible exploits in code because of it. For > example, imagine a class: > > -> I think extending and delegating to the super class was the case a long time ago but was causing all sorts of problems. I agree that proxying and delegating to the real class also causes other problems. We should ask them what caused then to pick the later. Doing that, we also need to be cautious about final methods. class Foo { > public final production; > Foo(boolean production) { this.production = production; } // Only call > from unit test > public Foo() { this(true); } > public void doSomethingSensible() { > if (production) { doSecurityCheck(); } > // do something ... > } > } > > With Objenesis, you can create an instance of the above class and set it > to test mode with the production field being false as the default value. In > contrast, for the case of serialization, I already argued that support for > serializable types should probably go into MethodHandles.Lookup similar to > how defineClass was added. This does not render an additional security > threat, if Foo was serializable I could just synthetically create a > serialized vale of an instance with production being false and deserialize > this value in a production environment. On a side note, Spring would create > a class with the above field value automatically if Foo was a bean, making > this exploit very easy even when using a security manager as the proxy > instance is normally available to all sorts of code. > > Finally, I disagree with the argument that this sort of API should be > accessible to "standard code" only shielded by a security manager because > this feature is useful today. The module system becomes useless the moment > you punch even a little hole into it. Therefore, I find it important to > remove sun.misc.Unsafe as quickly as possible as the module system can be > easily circumvented until it is removed and since the JPMS only show its > full benefit once this is no longer possible. But in order to keep testing > frameworks that have a good reason to break module boundaries up and > running, I think adding a test modules is a reasonable compromise. As an > additional benefit, it would allow libraries like Mockito that should not > be used in production to self-identify. > -> I don't see the breach in JPMS in particular. Reflection is restricted. Creating a class without calling a constructor is reflection. -> Using mockito in production is weird. Using Mockito in a stubbing server can make sense. But you can advocate it requires to add some parameters to the JVM to make it work. > > One could even argue that this ability to instantiate instance is nothing > that needs to be exposed to a Java agent so maybe this functionality could > also be part of the mentioned test module instead of being a part of > instrumentation. On the other hand, if there is a use case not being > considered for a Java agent, this agent might be tempted to open > jdk.internal.misc to gain this access, one might therefore just decide to > add the method to Instrumentation to offer an easy migration path to > applications that aim to migrate their code with a minimal delta. > > Best regards, Rafael > > 2018-04-03 21:26 GMT+02:00 Henri Tremblay : > >> Hi Rafael, >> >> I don't like the idea to have to explicitly load a module to create mocks. >> >> Because Unsafe.allocateInstance is of course used to create mock but also >> proxies. For instance, Spring uses it extensively through Objenesis. And >> different frameworks doing serialization. >> >> Also, It means Gradle and Maven would need to be updated to load the >> module automatically in test context. And all IDE. A lot of annoyance for >> not much benefit since everybody is currently happy. >> >> I've been advocating that if creating a class without calling a >> constructor is useless in multiple contexts, it should be easily available >> through the API. I don't know exactly where to put it but it should be >> >> - Available to all (java.base) >> - Protected by the security manager >> >> For completeness, there are 4 ways to create a class without calling a >> constructor right now that I'm aware of: >> >> - Unsafe.allocateInstance >> - sun.reflect.ReflectionFactory.newConstructorForSerialization (my >> favorite because it's the fastest one) >> - Generate an extending class forgetting to call the super >> constructor (so it's not exactly that same class that is instantiated). It >> requires -Xverify:none >> - Generate a class extending MagicAccessorImpl that will then >> instantiates the wanted class but calling the wrong constructor >> >> Regards, >> Henri >> >> >> >> On 1 April 2018 at 17:02, Rafael Winterhalter >> wrote: >> >>> Hello, >>> >>> I am the/a maintainer of the libraries Byte Buddy, cglib and Mockito with >>> countless dependencies upstream and I wanted to give a summary of >>> adopting >>> the JPMS and migrating away from sun.misc.Unsafe. >>> >>> 1. Java agents cannot define auxiliary classes. >>> >>> Byte Buddy does support the JPMS fully, however, it still relies on >>> sun.misc.Unsafe::defineClass for its Java agent API and currently breaks >>> on >>> Java 11 as this method was removed in a recent EA build. The reason for >>> using Unsafe is that many instrumentations need to define auxiliary >>> classes >>> to aid an instrumentation similar to javac which sometimes needs to >>> define >>> anonymous classes or even synthetic classes. For example, if a Java agent >>> wants to register an event listener to some framework, such listeners >>> often >>> declare multiple methods what makes it impossible to fullfil the listener >>> contract using a lambda expression. Instead, one typically injects an >>> additional class into the same package as the instrumented class. In this >>> case, it is not possible to use MethodHandles.Lookup::defineClass as the >>> class file transformer does not necessarily have private access to the >>> lookup of the instrumented class. >>> >>> The current workarounds are: >>> >>> a) Open the package jdk.internal.misc to gain access to this package's >>> Unsafe class. This can be done via Instrumentation::redefineModule. >>> b) Open the java.lang package to access ClassLoader via reflection. >>> c) Open the java.lang package to access the internal lookup with global >>> access rights. >>> >>> Of these solutions only (b) relies on standard API and is guaranteed to >>> function in the future but the solution still feels hacky and does not >>> work >>> for instrumentations of classes on the bootstrap loader. Opening packages >>> also implies a risk of being applied carelessly since opening the package >>> to the agent's module most likely opens the package to the unnamed module >>> of the system class loader what invites to breaches of the JPMS >>> encapsulation by code that does not ship with the agent. >>> >>> To offer a better solution, I would like to suggest one of the following: >>> >>> a) Add a method defineClass(ClassLoader, byte[], ProtectionDomain) to the >>> Instrumentation interface that works similar to Unsafe::defineClass. This >>> would provide a very simple migration path. Since agents have access to >>> jdk.internal.misc, adding this method does not add any capabilities to >>> the >>> agent, it merley avoids using internal API that might change. >>> b) Supply a MethodHandles.Lookup instance to the >>> ClassFileTransformer::transform API where the instance represents the >>> instrumented class's access rights. This does however carry the risk of >>> invoking the lookupClass method which would either load the instrumented >>> class prematurely causing a circularity error or return an unexpected >>> value >>> such as null. Since the lookup API generally relies on loaded types, >>> there >>> are probably other problems such as invoking Lookup::privateLookupIn >>> before >>> all involved types are loaded. >>> >>> For the sake of simlicity and since easy migration paths make a quick >>> adoption easier, I would suggestion solution (a), also in the light that >>> quick and dirty migrations might still choose option (b) to save time and >>> also since (b) might cause problems when types are not yet loaded. >>> >>> 2. Java proxies cannot invoke default methods of proxied interfaces >>> >>> The Java proxy API does not currently allow the invocation of an >>> overridden >>> default method since >>> the InvocationHandler API only supplies an instance of >>> java.lang.reflection.Method. In Java 8, it was always possible to get >>> hold >>> of method handle of any method of the proxied interface and to specialize >>> the handle on the interface type to invoke the default implementation. >>> With >>> the JPMS, even if an interface type is exported, this same type might not >>> be opened to another module. This implies that if an InvocationHandler is >>> defined by this module to which the interface is exported, this module's >>> InvocationHandler cannot resolve a specialized method handle to a default >>> method of the proxied interface. As a matter of fact, such a call is >>> impossible in this scenario whereas the same call is possible if the >>> proxy >>> is implemented manually at compile time. >>> >>> As a solution, I suggest to provide an argument to the InvocationHandler >>> that represents a lookup instance with the access rights of the proxy >>> class. Using this lookup, the specialized method handles could be >>> resolved. >>> >>> 3. Mocking and serialization libraries still require >>> Unsafe::allocateInstance. >>> >>> For Mockito, it is required to instantiate any class without invoking a >>> constructor with potential side-effects. This is of course a grose >>> violation of Java's general contract for class instantiation but this >>> feature is very useful. >>> >>> Using a Java agent, it is already possible to emulate this feature >>> without >>> internal API by instrumenting all constructors of all classes in the >>> hierarchy of a mocked class by transforming all constructors into the >>> following pseudo-code: >>> >>> SomeConstructor(Void arg) { >>> if (MockitoBootHelper.THREAD_LOCAL.isMockInstantiatingMode()) { >>> super(null); // any constructor of the super class with default >>> values >>> for all arguments >>> } else { >>> // original constructor code... >>> } >>> } >>> >>> Before instantiating a mock, the thread local value that is held by a >>> bootstrap-loader injected class is set to true such that a side-effect >>> free >>> construction is achieved. >>> >>> This is of course too expensive and has negative effects on performance >>> due >>> to additional branching and JIT-byte code limits such that one would >>> rather >>> open jdk.internal.misc to access the Unsafe instantiation mechanism if a >>> Java agent is already available. >>> >>> However, mocking and serialization libraries are not typically loaded as >>> a >>> Java agent. Also, I think that the actual requirements are different. My >>> suggestion here is: >>> >>> a) For serialization libraries, I think that adding >>> MethodHandles.Lookup::newInstance(Class) with >>> standard deserialization mechanics would be sufficient. >>> b) For mocking libraries, this does not suffice as mocks can be of any >>> class. I understand that this breaks encapsulation but for unit tests, I >>> argue that the benefit of using these libraries outweights the benefit of >>> full encapsulation within a unit test. >>> >>> As Mockito is typically run within a build which uses a JDK, we could >>> attach to the current VM using the attachment API. Since Java 9, this is >>> no >>> longer possible to attach to the same VM but instead we start a helper >>> JVM >>> process that applies the attachment indirectly. Unfortunately, this is a >>> rather costly operation what is especially problematic when running a >>> single unit test. (The difference to this approach over Unsafe is about >>> half a second on average.) >>> >>> To overcome this, I would like to suggest to: >>> >>> a) Add a method Instrumentation::allocateInstance(Class). Java agents >>> can >>> already emulate this privilege as described above, this is therefore >>> merely >>> a convenience. >>> b) Add a module jdk.test to JDK 11 with a class >>> JavaTest::getInstrumentation that returns an instrumentation instance for >>> the current JVM. This module should not be resolved by default but only >>> when requiring it explicitly similar to the EE modules after Java 9. >>> >>> I think this solution carries two benefits: >>> >>> a) Test libraries like Mockito can only be used in a testing scope. We >>> experience regularly that Mockito is used in production environments. The >>> library is not meant for that and we warn and document that this is not >>> intended use but people regularly ignore this directive. By requiring >>> this >>> module, this form of abuse would no longer be possible and the JVM could >>> even give a meaningful error message if this use was intended. >>> b) The Instrumentation instance can be used for other meaningful testing >>> approaches. For example, Mockito offers an inline mock maker where the >>> mocking logic is inlined into a method body rather then creating a >>> subclass. This approach mainly targets final classes which have become >>> more >>> common especially since the Kotlin language became popular. To supply >>> this >>> alternative mock maker, Mockito attempts attaching an agent to the >>> current >>> VM (directly or indirectly, depending on the VM's version) which suffers >>> the additional costs for attaching an agent that I mentioned before. >>> >>> Thanks for reading this and I hope that you can consider these, as of >>> today, very practiced use cases. The JPMS migration has gone quite well, >>> I >>> find. With these outstanding problems, JDK 11 could be the first release >>> where a majority of Java programs would no longer need to rely on >>> internal >>> API. >>> >>> Best regards, Rafael >>> >> >> > From rafael.wth at gmail.com Fri Apr 6 17:20:52 2018 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Fri, 6 Apr 2018 19:20:52 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: Message-ID: As for final methods, this is a problem with delegating and subclassing proxies. For delegation, I would even claim that this is a bigger problem as the original code is invoked on an unprepared instance. With subclassing, the final method is simply invoked in its non-proxied form what should retain consistent state. Also, subclassing proxies can emulate self-invocation without proxying by setting a flag on the instance that is reset after the proxied instance completes its super method call. Finally, subclassing proxies avoid confusion when using instances as monitors etc. The problem with creating instances without invoking a constructor that I see when using the lookup facility, for instance, is that this lookup would need access to the entire class hierarchy. If I defined a class Foo extends Bar extends Object and Foo and Bar resided in different packages and I wanted to invoke an instance of Foo without a constructor call, my lookup might have the right access for Foo but not for Bar. Should this then require another lookup to skip Bar's constructor as well. And if not, could I not just define a subclass of Foo, get a handle for it and now instantiate a subclass of Foo that can be used as an instance of Foo? This would break both Foo's and Bar's encapsulation without enforcing any restriction. I still find instantiating non-serializable classes without constructor calls not meaningful for production environments, at least I could not think of a good use case. For a stubbing server, I claim that the mocking must work on a higher level then an object instance. Since such a server needs deployment, I would also call this server a production application even if it might never run in a production environment. As for application tests, it can make sense for mocking and other unit testing frameworks and I think a Java agent would be a good faciliator as owners of an Instrumentation instance already can emulate this behavior as I had described above. Best regards, Rafael 2018-04-06 5:37 GMT+02:00 Henri Tremblay : > Answers inline > > On 4 April 2018 at 08:05, Rafael Winterhalter > wrote: > >> Hei Henri, >> >> for Maven, Gradle and IDEs I think it is sensible to expect that these >> runners would resolve such a test module unless it is explicitly specified >> otherwise. Already today, those runners patch the module under test in >> order to include the test code. I would expect of these runners to adopt >> similarly to a test module as they adopted to the module system in general. >> >> -> Yes. I agree. > > >> As for Spring's use case, I believe that the approach was chosen such >> that class proxies resemble interface proxies as much as possible. In >> general however, I think that Spring could just generate a subclass proxy >> and delegate to the overridden methods instead of delegating to an instance >> of the proxied class. If the proxy class mimics the proxied class's >> constructors' signatures, this proxy can be instantiated exactly like the >> proxied class would have been. This way, using Objenesis and the like is no >> longer necessary and I argue that one should avoid exposing this >> functionality if there is a reasonable alternative that does not require >> any derivation from Javas programming model. As a matter of fact, I often >> find developers confused about how fields are not set on such Spring >> proxies and I even found possible exploits in code because of it. For >> example, imagine a class: >> >> -> I think extending and delegating to the super class was the case a > long time ago but was causing all sorts of problems. I agree that proxying > and delegating to the real class also causes other problems. We should ask > them what caused then to pick the later. Doing that, we also need to be > cautious about final methods. > > class Foo { >> public final production; >> Foo(boolean production) { this.production = production; } // Only call >> from unit test >> public Foo() { this(true); } >> public void doSomethingSensible() { >> if (production) { doSecurityCheck(); } >> // do something ... >> } >> } >> >> With Objenesis, you can create an instance of the above class and set it >> to test mode with the production field being false as the default value. In >> contrast, for the case of serialization, I already argued that support for >> serializable types should probably go into MethodHandles.Lookup similar to >> how defineClass was added. This does not render an additional security >> threat, if Foo was serializable I could just synthetically create a >> serialized vale of an instance with production being false and deserialize >> this value in a production environment. On a side note, Spring would create >> a class with the above field value automatically if Foo was a bean, making >> this exploit very easy even when using a security manager as the proxy >> instance is normally available to all sorts of code. >> >> Finally, I disagree with the argument that this sort of API should be >> accessible to "standard code" only shielded by a security manager because >> this feature is useful today. The module system becomes useless the moment >> you punch even a little hole into it. Therefore, I find it important to >> remove sun.misc.Unsafe as quickly as possible as the module system can be >> easily circumvented until it is removed and since the JPMS only show its >> full benefit once this is no longer possible. But in order to keep testing >> frameworks that have a good reason to break module boundaries up and >> running, I think adding a test modules is a reasonable compromise. As an >> additional benefit, it would allow libraries like Mockito that should not >> be used in production to self-identify. >> > > -> I don't see the breach in JPMS in particular. Reflection is > restricted. Creating a class without calling a constructor is reflection. > -> Using mockito in production is weird. Using Mockito in a stubbing > server can make sense. But you can advocate it requires to add some > parameters to the JVM to make it work. > >> >> One could even argue that this ability to instantiate instance is nothing >> that needs to be exposed to a Java agent so maybe this functionality could >> also be part of the mentioned test module instead of being a part of >> instrumentation. On the other hand, if there is a use case not being >> considered for a Java agent, this agent might be tempted to open >> jdk.internal.misc to gain this access, one might therefore just decide to >> add the method to Instrumentation to offer an easy migration path to >> applications that aim to migrate their code with a minimal delta. >> >> Best regards, Rafael >> >> 2018-04-03 21:26 GMT+02:00 Henri Tremblay : >> >>> Hi Rafael, >>> >>> I don't like the idea to have to explicitly load a module to create >>> mocks. >>> >>> Because Unsafe.allocateInstance is of course used to create mock but >>> also proxies. For instance, Spring uses it extensively through Objenesis. >>> And different frameworks doing serialization. >>> >>> Also, It means Gradle and Maven would need to be updated to load the >>> module automatically in test context. And all IDE. A lot of annoyance for >>> not much benefit since everybody is currently happy. >>> >>> I've been advocating that if creating a class without calling a >>> constructor is useless in multiple contexts, it should be easily available >>> through the API. I don't know exactly where to put it but it should be >>> >>> - Available to all (java.base) >>> - Protected by the security manager >>> >>> For completeness, there are 4 ways to create a class without calling a >>> constructor right now that I'm aware of: >>> >>> - Unsafe.allocateInstance >>> - sun.reflect.ReflectionFactory.newConstructorForSerialization (my >>> favorite because it's the fastest one) >>> - Generate an extending class forgetting to call the super >>> constructor (so it's not exactly that same class that is instantiated). It >>> requires -Xverify:none >>> - Generate a class extending MagicAccessorImpl that will then >>> instantiates the wanted class but calling the wrong constructor >>> >>> Regards, >>> Henri >>> >>> >>> >>> On 1 April 2018 at 17:02, Rafael Winterhalter >>> wrote: >>> >>>> Hello, >>>> >>>> I am the/a maintainer of the libraries Byte Buddy, cglib and Mockito >>>> with >>>> countless dependencies upstream and I wanted to give a summary of >>>> adopting >>>> the JPMS and migrating away from sun.misc.Unsafe. >>>> >>>> 1. Java agents cannot define auxiliary classes. >>>> >>>> Byte Buddy does support the JPMS fully, however, it still relies on >>>> sun.misc.Unsafe::defineClass for its Java agent API and currently >>>> breaks on >>>> Java 11 as this method was removed in a recent EA build. The reason for >>>> using Unsafe is that many instrumentations need to define auxiliary >>>> classes >>>> to aid an instrumentation similar to javac which sometimes needs to >>>> define >>>> anonymous classes or even synthetic classes. For example, if a Java >>>> agent >>>> wants to register an event listener to some framework, such listeners >>>> often >>>> declare multiple methods what makes it impossible to fullfil the >>>> listener >>>> contract using a lambda expression. Instead, one typically injects an >>>> additional class into the same package as the instrumented class. In >>>> this >>>> case, it is not possible to use MethodHandles.Lookup::defineClass as >>>> the >>>> class file transformer does not necessarily have private access to the >>>> lookup of the instrumented class. >>>> >>>> The current workarounds are: >>>> >>>> a) Open the package jdk.internal.misc to gain access to this package's >>>> Unsafe class. This can be done via Instrumentation::redefineModule. >>>> b) Open the java.lang package to access ClassLoader via reflection. >>>> c) Open the java.lang package to access the internal lookup with global >>>> access rights. >>>> >>>> Of these solutions only (b) relies on standard API and is guaranteed to >>>> function in the future but the solution still feels hacky and does not >>>> work >>>> for instrumentations of classes on the bootstrap loader. Opening >>>> packages >>>> also implies a risk of being applied carelessly since opening the >>>> package >>>> to the agent's module most likely opens the package to the unnamed >>>> module >>>> of the system class loader what invites to breaches of the JPMS >>>> encapsulation by code that does not ship with the agent. >>>> >>>> To offer a better solution, I would like to suggest one of the >>>> following: >>>> >>>> a) Add a method defineClass(ClassLoader, byte[], ProtectionDomain) to >>>> the >>>> Instrumentation interface that works similar to Unsafe::defineClass. >>>> This >>>> would provide a very simple migration path. Since agents have access to >>>> jdk.internal.misc, adding this method does not add any capabilities to >>>> the >>>> agent, it merley avoids using internal API that might change. >>>> b) Supply a MethodHandles.Lookup instance to the >>>> ClassFileTransformer::transform API where the instance represents the >>>> instrumented class's access rights. This does however carry the risk of >>>> invoking the lookupClass method which would either load the instrumented >>>> class prematurely causing a circularity error or return an unexpected >>>> value >>>> such as null. Since the lookup API generally relies on loaded types, >>>> there >>>> are probably other problems such as invoking Lookup::privateLookupIn >>>> before >>>> all involved types are loaded. >>>> >>>> For the sake of simlicity and since easy migration paths make a quick >>>> adoption easier, I would suggestion solution (a), also in the light that >>>> quick and dirty migrations might still choose option (b) to save time >>>> and >>>> also since (b) might cause problems when types are not yet loaded. >>>> >>>> 2. Java proxies cannot invoke default methods of proxied interfaces >>>> >>>> The Java proxy API does not currently allow the invocation of an >>>> overridden >>>> default method since >>>> the InvocationHandler API only supplies an instance of >>>> java.lang.reflection.Method. In Java 8, it was always possible to get >>>> hold >>>> of method handle of any method of the proxied interface and to >>>> specialize >>>> the handle on the interface type to invoke the default implementation. >>>> With >>>> the JPMS, even if an interface type is exported, this same type might >>>> not >>>> be opened to another module. This implies that if an InvocationHandler >>>> is >>>> defined by this module to which the interface is exported, this module's >>>> InvocationHandler cannot resolve a specialized method handle to a >>>> default >>>> method of the proxied interface. As a matter of fact, such a call is >>>> impossible in this scenario whereas the same call is possible if the >>>> proxy >>>> is implemented manually at compile time. >>>> >>>> As a solution, I suggest to provide an argument to the InvocationHandler >>>> that represents a lookup instance with the access rights of the proxy >>>> class. Using this lookup, the specialized method handles could be >>>> resolved. >>>> >>>> 3. Mocking and serialization libraries still require >>>> Unsafe::allocateInstance. >>>> >>>> For Mockito, it is required to instantiate any class without invoking a >>>> constructor with potential side-effects. This is of course a grose >>>> violation of Java's general contract for class instantiation but this >>>> feature is very useful. >>>> >>>> Using a Java agent, it is already possible to emulate this feature >>>> without >>>> internal API by instrumenting all constructors of all classes in the >>>> hierarchy of a mocked class by transforming all constructors into the >>>> following pseudo-code: >>>> >>>> SomeConstructor(Void arg) { >>>> if (MockitoBootHelper.THREAD_LOCAL.isMockInstantiatingMode()) { >>>> super(null); // any constructor of the super class with default >>>> values >>>> for all arguments >>>> } else { >>>> // original constructor code... >>>> } >>>> } >>>> >>>> Before instantiating a mock, the thread local value that is held by a >>>> bootstrap-loader injected class is set to true such that a side-effect >>>> free >>>> construction is achieved. >>>> >>>> This is of course too expensive and has negative effects on performance >>>> due >>>> to additional branching and JIT-byte code limits such that one would >>>> rather >>>> open jdk.internal.misc to access the Unsafe instantiation mechanism if a >>>> Java agent is already available. >>>> >>>> However, mocking and serialization libraries are not typically loaded >>>> as a >>>> Java agent. Also, I think that the actual requirements are different. My >>>> suggestion here is: >>>> >>>> a) For serialization libraries, I think that adding >>>> MethodHandles.Lookup::newInstance(Class) with >>>> standard deserialization mechanics would be sufficient. >>>> b) For mocking libraries, this does not suffice as mocks can be of any >>>> class. I understand that this breaks encapsulation but for unit tests, I >>>> argue that the benefit of using these libraries outweights the benefit >>>> of >>>> full encapsulation within a unit test. >>>> >>>> As Mockito is typically run within a build which uses a JDK, we could >>>> attach to the current VM using the attachment API. Since Java 9, this >>>> is no >>>> longer possible to attach to the same VM but instead we start a helper >>>> JVM >>>> process that applies the attachment indirectly. Unfortunately, this is a >>>> rather costly operation what is especially problematic when running a >>>> single unit test. (The difference to this approach over Unsafe is about >>>> half a second on average.) >>>> >>>> To overcome this, I would like to suggest to: >>>> >>>> a) Add a method Instrumentation::allocateInstance(Class). Java agents >>>> can >>>> already emulate this privilege as described above, this is therefore >>>> merely >>>> a convenience. >>>> b) Add a module jdk.test to JDK 11 with a class >>>> JavaTest::getInstrumentation that returns an instrumentation instance >>>> for >>>> the current JVM. This module should not be resolved by default but only >>>> when requiring it explicitly similar to the EE modules after Java 9. >>>> >>>> I think this solution carries two benefits: >>>> >>>> a) Test libraries like Mockito can only be used in a testing scope. We >>>> experience regularly that Mockito is used in production environments. >>>> The >>>> library is not meant for that and we warn and document that this is not >>>> intended use but people regularly ignore this directive. By requiring >>>> this >>>> module, this form of abuse would no longer be possible and the JVM could >>>> even give a meaningful error message if this use was intended. >>>> b) The Instrumentation instance can be used for other meaningful testing >>>> approaches. For example, Mockito offers an inline mock maker where the >>>> mocking logic is inlined into a method body rather then creating a >>>> subclass. This approach mainly targets final classes which have become >>>> more >>>> common especially since the Kotlin language became popular. To supply >>>> this >>>> alternative mock maker, Mockito attempts attaching an agent to the >>>> current >>>> VM (directly or indirectly, depending on the VM's version) which suffers >>>> the additional costs for attaching an agent that I mentioned before. >>>> >>>> Thanks for reading this and I hope that you can consider these, as of >>>> today, very practiced use cases. The JPMS migration has gone quite >>>> well, I >>>> find. With these outstanding problems, JDK 11 could be the first release >>>> where a majority of Java programs would no longer need to rely on >>>> internal >>>> API. >>>> >>>> Best regards, Rafael >>>> >>> >>> >> > From javalists at cbfiddle.com Sat Apr 7 16:50:51 2018 From: javalists at cbfiddle.com (Alan Snyder) Date: Sat, 7 Apr 2018 09:50:51 -0700 Subject: a program that fails only when run as a macOS bundled app Message-ID: I?ve run into a strange situation, a tiny Java Swing program that works when run from the command line (java -jar) but fails when run as a macOS bundled app created by the packager. I am using Java 10: java version "10" 2018-03-20 Java(TM) SE Runtime Environment 18.3 (build 10+46) Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10+46, mixed mode) The specific error appears to involve creating a window with an unsupported set of style bits. A similar error is described in JDK-8181476 [1]. NSWindow logs an assertion error. The problem does not arise in Java 8, probably because it uses a different set of style bits. Does anyone have an idea why the behavior would be different for the bundled app vs the command line? Anyone willing to take a look at it? Alan [1] https://bugs.openjdk.java.net/browse/JDK-8181476 package test; import javax.swing.JFrame; import javax.swing.SwingUtilities; public class Test { public Test() { // This frame is always displayed: JFrame fr1 = new JFrame("Test 1"); fr1.getRootPane().putClientProperty("Window.style", "small"); fr1.setResizable(false); fr1.setBounds(0, 0, 200, 200); fr1.setVisible(true); // This frame fails to display when run as a Java 10 bundled app: JFrame fr2 = new JFrame("Test 2"); fr2.getRootPane().putClientProperty("Window.style", "small"); fr2.setBounds(300, 0, 200, 200); fr2.setVisible(true); } public static void main(String[] args) { SwingUtilities.invokeLater(new Runnable() { public void run() { new Test(); } }); } } From Alan.Bateman at oracle.com Mon Apr 9 07:33:27 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 9 Apr 2018 08:33:27 +0100 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: Message-ID: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> On 01/04/2018 22:02, Rafael Winterhalter wrote: > : > > 1. Java agents cannot define auxiliary classes. > > : > The reason for > using Unsafe is that many instrumentations need to define auxiliary classes > to aid an instrumentation similar to javac which sometimes needs to define > anonymous classes or even synthetic classes. For example, if a Java agent > wants to register an event listener to some framework, such listeners often > declare multiple methods what makes it impossible to fullfil the listener > contract using a lambda expression. Instead, one typically injects an > additional class into the same package as the instrumented class. This seems a reasonable requirement. As you know, JSR-163 created this API (and JVM TI) for tools to instrument code in mostly benign ways where any additional agent provided helper classes are made visibility via the appendToXXXClassLoaderSearch methods. I don't think the use-case of dynamically generated helper classes came up, I don't recall it coming up on serviceability-dev in the intervening years either. In any case, I think there should be a way to support this scenario, it amounts to a ClassFileTransformer providing the class bytes of additional classes to be defined in the same runtime package as the class being loaded or transformed. There are a number of API choices and it's probably best if we bring proposals to serviceability-dev as that is where this API is maintained. > : > > 2. Java proxies cannot invoke default methods of proxied interfaces > > The Java proxy API does not currently allow the invocation of an overridden > default method since > the InvocationHandler API only supplies an instance of > java.lang.reflection.Method. The issue of Proxies and default methods has come up on core-libs-dev a few times. In the mean-time, JEP 274 added support for MethodHandles that bind to non-abstract methods in interfaces. I just double checked and I can create a proxy where the invocation handler creates a method handle to the default method, binds it to proxy, and invokes it. I also checked the case where the interface is public in an exported package of a named module. Can you say a bit more, or provide an example, where you run into issues? I'm wondering if you are running into an accessibility issue or something else. -Alan From scolebourne at joda.org Mon Apr 9 10:56:20 2018 From: scolebourne at joda.org (Stephen Colebourne) Date: Mon, 9 Apr 2018 11:56:20 +0100 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> References: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> Message-ID: On 9 April 2018 at 08:33, Alan Bateman wrote: >> 2. Java proxies cannot invoke default methods of proxied interfaces >> >> The Java proxy API does not currently allow the invocation of an >> overridden >> default method since >> the InvocationHandler API only supplies an instance of >> java.lang.reflection.Method. > > The issue of Proxies and default methods has come up on core-libs-dev a few > times. I hit this problem just today but am on Java 8, so Java 9 solutions are no good. The standard Java 8 workaround requires setAccessible. I found an alternative that is useful in some cases, but was worth documenting (as I didn't find it described anywhere else): https://stackoverflow.com/a/49730826/38896 I do think Proxy should be enhanced to handle default methods without use of MethodHandle. Proxy is more of an entry level tool, whereas MH is lower level. Requiring MH to solve default methods seems wrong for the typical user of Proxy. Stephen From rafael.wth at gmail.com Mon Apr 9 19:48:42 2018 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Mon, 9 Apr 2018 21:48:42 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> References: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> Message-ID: Hei Alan, maybe I am doing it wrong but this is my example. I created a module with two interfaces bar.Bar and qux.Qux that both define a default method String foo() { return "foo"; }. The module exports bar and exports and opens qux. >From another module that reads that first module I run the following code: package main; import bar.Bar; import qux.Qux; import java.lang.invoke.MethodHandles; import java.lang.invoke.MethodType; import java.lang.reflect.Proxy; import java.util.function.Consumer; public class Main { public static void main(String[] args) { test(Foo.class, Foo::foo); // works: interface in same module test(Bar.class, Bar::foo); // works not: interface in other module, exported test(Qux.class, Qux::foo); // works: interface in other module, exported and opened test(Bar.class, new Bar() { // works: explicit proxy @Override public String foo() { return Bar.super.foo(); } }, Bar::foo); test(Bar.class, new Bar() { // works not: explicit proxy with method handle @Override public String foo() { try { return (String) MethodHandles.lookup().findSpecial( Bar.class, "foo", MethodType.methodType(String.class, new Class[0]), Bar.class ).bindTo(this).invokeWithArguments(); } catch (Throwable throwable) { throw new RuntimeException(throwable); } } }, Bar::foo); } private static void test(Class iface, Consumer consumer) { Object instance = Proxy.newProxyInstance( Bar.class.getClassLoader(), new Class[]{iface}, (proxy, method, arguments) -> MethodHandles.privateLookupIn(iface, MethodHandles.lookup()).findSpecial( iface, "foo", MethodType.methodType(String.class, new Class[0]), iface ).bindTo(proxy).invokeWithArguments() ); test(iface, iface.cast(instance), consumer); } private static void test(Class iface, T instance, Consumer consumer) { try { consumer.accept(instance); System.out.println("Could invoke special method for " + iface); } catch (Throwable throwable) { System.out.println("Could NOT invoke special method for " + iface); } } } >From the code comments, some approaches work and some do not. What would I need to do differently? Thank you and best regards, Rafael 2018-04-09 9:33 GMT+02:00 Alan Bateman : > On 01/04/2018 22:02, Rafael Winterhalter wrote: > >> : >> >> 1. Java agents cannot define auxiliary classes. >> >> : >> The reason for >> using Unsafe is that many instrumentations need to define auxiliary >> classes >> to aid an instrumentation similar to javac which sometimes needs to define >> anonymous classes or even synthetic classes. For example, if a Java agent >> wants to register an event listener to some framework, such listeners >> often >> declare multiple methods what makes it impossible to fullfil the listener >> contract using a lambda expression. Instead, one typically injects an >> additional class into the same package as the instrumented class. >> > This seems a reasonable requirement. As you know, JSR-163 created this API > (and JVM TI) for tools to instrument code in mostly benign ways where any > additional agent provided helper classes are made visibility via the > appendToXXXClassLoaderSearch methods. I don't think the use-case of > dynamically generated helper classes came up, I don't recall it coming up > on serviceability-dev in the intervening years either. In any case, I think > there should be a way to support this scenario, it amounts to a > ClassFileTransformer providing the class bytes of additional classes to be > defined in the same runtime package as the class being loaded or > transformed. There are a number of API choices and it's probably best if we > bring proposals to serviceability-dev as that is where this API is > maintained. > > > : >> >> 2. Java proxies cannot invoke default methods of proxied interfaces >> >> The Java proxy API does not currently allow the invocation of an >> overridden >> default method since >> the InvocationHandler API only supplies an instance of >> java.lang.reflection.Method. >> > The issue of Proxies and default methods has come up on core-libs-dev a > few times. In the mean-time, JEP 274 added support for MethodHandles that > bind to non-abstract methods in interfaces. I just double checked and I can > create a proxy where the invocation handler creates a method handle to the > default method, binds it to proxy, and invokes it. I also checked the case > where the interface is public in an exported package of a named module. Can > you say a bit more, or provide an example, where you run into issues? I'm > wondering if you are running into an accessibility issue or something else. > > -Alan > > From Alan.Bateman at oracle.com Tue Apr 10 09:54:52 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 10 Apr 2018 10:54:52 +0100 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> Message-ID: <2b141a11-ebef-ca86-d37f-e9060d309e82@oracle.com> On 09/04/2018 20:48, Rafael Winterhalter wrote: > Hei Alan, > maybe I am doing it wrong but this is my example. I created a module > with two interfaces bar.Bar and qux.Qux that both define a default > method String foo() { return "foo"; }. The module exports bar and > exports and opens qux. > From another module that reads that first module I run the following code: Thanks for the test. I think the issue is that findSpecial can never work when invoked on a Lookup to a lookup class in a named module and the proposed caller class for invokespecial (specialCaller) is in a different module. It's not an issue when the lookup class in an unnamed module, or when the lookup class in a named module and specialCaller is in the same module. I'm sure John will jump in but I think if JDK-8173978 is implemented then it will go a long way to address this anomaly. -Alan [1] https://bugs.openjdk.java.net/browse/JDK-8173978 From rafael.wth at gmail.com Tue Apr 10 11:15:59 2018 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Tue, 10 Apr 2018 13:15:59 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: <2b141a11-ebef-ca86-d37f-e9060d309e82@oracle.com> References: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> <2b141a11-ebef-ca86-d37f-e9060d309e82@oracle.com> Message-ID: Thanks for the reference. As for the proxy API, I think that this should be reworked anyways as the lookup on each call would be rather expensive. I also see the other issues that I mentioned as more pressing. Thank you and best regards, Rafael 2018-04-10 11:54 GMT+02:00 Alan Bateman : > On 09/04/2018 20:48, Rafael Winterhalter wrote: > >> Hei Alan, >> maybe I am doing it wrong but this is my example. I created a module with >> two interfaces bar.Bar and qux.Qux that both define a default method String >> foo() { return "foo"; }. The module exports bar and exports and opens qux. >> From another module that reads that first module I run the following code: >> > Thanks for the test. > > I think the issue is that findSpecial can never work when invoked on a > Lookup to a lookup class in a named module and the proposed caller class > for invokespecial (specialCaller) is in a different module. It's not an > issue when the lookup class in an unnamed module, or when the lookup class > in a named module and specialCaller is in the same module. > > I'm sure John will jump in but I think if JDK-8173978 is implemented then > it will go a long way to address this anomaly. > > -Alan > > [1] https://bugs.openjdk.java.net/browse/JDK-8173978 > From Alan.Bateman at oracle.com Wed Apr 11 07:26:44 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 11 Apr 2018 08:26:44 +0100 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> <2b141a11-ebef-ca86-d37f-e9060d309e82@oracle.com> Message-ID: <7aee3cd3-dc2e-3fc6-5cb6-0192d3f17943@oracle.com> On 10/04/2018 12:15, Rafael Winterhalter wrote: > Thanks for the reference. > > As for the proxy API, I think that this should be reworked anyways as > the lookup on each call would be rather expensive. In the mean-time, I think you should be able to make progress with findSpecial for the common case where the interface is public and in an exported package. For this case, the proxy will be generated into an unnamed module (the "Package and Module Membership of Proxy Class" section of the Proxy API spec has all the wonderful details on this). The proxy dispatches to your invocation handler, module "main" in your test. The invocation handler can invoke Module::addReads to update module "main" to read the proxy's module. It can then create a full power Lookup on the proxy class and invoke findSpecial with the proxy class ("iface" should work too) as the specialCaller. Bind that to the proxy instance and perform the invoke as you are doing already. -Alan. From rafael.wth at gmail.com Wed Apr 11 20:07:16 2018 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Wed, 11 Apr 2018 22:07:16 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: <7aee3cd3-dc2e-3fc6-5cb6-0192d3f17943@oracle.com> References: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> <2b141a11-ebef-ca86-d37f-e9060d309e82@oracle.com> <7aee3cd3-dc2e-3fc6-5cb6-0192d3f17943@oracle.com> Message-ID: I do not think that this is possible. If the module containing the interface does not open a package, I cannot change the privileges of the main module such that I can resolve a method handle for invoking the special invocation. I just tried this out too and I did not find a way, could you suggest how to change my code for being able to do so? Best regards, Rafael 2018-04-11 9:26 GMT+02:00 Alan Bateman : > On 10/04/2018 12:15, Rafael Winterhalter wrote: > >> Thanks for the reference. >> >> As for the proxy API, I think that this should be reworked anyways as the >> lookup on each call would be rather expensive. >> > In the mean-time, I think you should be able to make progress with > findSpecial for the common case where the interface is public and in an > exported package. > > For this case, the proxy will be generated into an unnamed module (the > "Package and Module Membership of Proxy Class" section of the Proxy API > spec has all the wonderful details on this). The proxy dispatches to your > invocation handler, module "main" in your test. The invocation handler can > invoke Module::addReads to update module "main" to read the proxy's module. > It can then create a full power Lookup on the proxy class and invoke > findSpecial with the proxy class ("iface" should work too) as the > specialCaller. Bind that to the proxy instance and perform the invoke as > you are doing already. > > -Alan. > > > From Alan.Bateman at oracle.com Thu Apr 12 08:40:28 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 12 Apr 2018 09:40:28 +0100 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> <2b141a11-ebef-ca86-d37f-e9060d309e82@oracle.com> <7aee3cd3-dc2e-3fc6-5cb6-0192d3f17943@oracle.com> Message-ID: On 11/04/2018 21:07, Rafael Winterhalter wrote: > I do not think that this is possible. If the module containing the > interface does not open a package, I cannot change the privileges of > the main module such that I can resolve a method handle for invoking > the special invocation. > > I just tried this out too and I did not find a way, could you suggest > how to change my code for being able to do so? If the interface is public in an exported package (no need for the package to be open) then the proxy will be generated into the unnamed module. So easy to get a Lookup to the proxy class and you can use this as the special caller. Can you change your invocation handler to the following and try it: Class proxyClass = proxy.getClass(); Main.class.getModule().addReads(proxyClass.getModule()); Lookup lookup = MethodHandles.privateLookupIn(proxyClass, MethodHandles.lookup()); MethodType mt = MethodType.methodType(String.class); return lookup.findSpecial(iface, "foo", mt, proxyClass).bindTo(proxy).invokeWithArguments(); -Alan From rafael.wth at gmail.com Thu Apr 12 16:06:13 2018 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Thu, 12 Apr 2018 18:06:13 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> <2b141a11-ebef-ca86-d37f-e9060d309e82@oracle.com> <7aee3cd3-dc2e-3fc6-5cb6-0192d3f17943@oracle.com> Message-ID: I have not thought of that but you are of course right, that works. The solution is however far from pretty as it needs a case basis since the resolution of the proxy class is only possible if the proxy is in fact loaded into an unnamed module what is not always the case. This does however cover all cases for Java 9 and Java 10 proxies. I do however hope Java 11 can ship with something more convenient then this: https://gist.github.com/raphw/c1faf2f40e80afce6f13511098cfb90f Thanks for guiding me through that! Best regards, Rafael 2018-04-12 10:40 GMT+02:00 Alan Bateman : > On 11/04/2018 21:07, Rafael Winterhalter wrote: > >> I do not think that this is possible. If the module containing the >> interface does not open a package, I cannot change the privileges of the >> main module such that I can resolve a method handle for invoking the >> special invocation. >> >> I just tried this out too and I did not find a way, could you suggest how >> to change my code for being able to do so? >> > If the interface is public in an exported package (no need for the package > to be open) then the proxy will be generated into the unnamed module. So > easy to get a Lookup to the proxy class and you can use this as the special > caller. Can you change your invocation handler to the following and try it: > > Class proxyClass = proxy.getClass(); > Main.class.getModule().addReads(proxyClass.getModule()); > Lookup lookup = MethodHandles.privateLookupIn(proxyClass, > MethodHandles.lookup()); > MethodType mt = MethodType.methodType(String.class); > return lookup.findSpecial(iface, "foo", mt, proxyClass).bindTo(proxy).invo > keWithArguments(); > > -Alan > From doug.simon at oracle.com Thu Apr 12 18:24:33 2018 From: doug.simon at oracle.com (Doug Simon) Date: Thu, 12 Apr 2018 20:24:33 +0200 Subject: RFR: 8187490: HotSpotRuntimeMBean should be moved to Graal management module Message-ID: <324AC8E7-9737-4DB9-875E-A7FDB551D75E@oracle.com> Please review this change that removes the existing Graal service provider for hooking into the Platform MBean Server and makes jdk.internal.vm.compiler.management an upgradeable module. Please refer to https://bugs.openjdk.java.net/browse/JDK-8187490?focusedCommentId=14170942&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14170942 for discussion on the latter point. The Graal changes that dynamically register an MBean for accessing Graal will be part of a subsequent Graal update. http://cr.openjdk.java.net/~dnsimon/8187490/ -Doug From mandy.chung at oracle.com Fri Apr 13 04:04:22 2018 From: mandy.chung at oracle.com (mandy chung) Date: Fri, 13 Apr 2018 12:04:22 +0800 Subject: RFR: 8187490: HotSpotRuntimeMBean should be moved to Graal management module In-Reply-To: <324AC8E7-9737-4DB9-875E-A7FDB551D75E@oracle.com> References: <324AC8E7-9737-4DB9-875E-A7FDB551D75E@oracle.com> Message-ID: <1cc549db-fc60-d0e3-8405-17d23701288b@oracle.com> On 4/13/18 2:24 AM, Doug Simon wrote: > Please review this change that removes the existing Graal service provider for hooking into the Platform MBean Server and makes jdk.internal.vm.compiler.management an upgradeable module. > > Please refer to https://bugs.openjdk.java.net/browse/JDK-8187490?focusedCommentId=14170942&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14170942 for discussion on the latter point. > > The Graal changes that dynamically register an MBean for accessing Graal will be part of a subsequent Graal update. > > http://cr.openjdk.java.net/~dnsimon/8187490/ > > This looks okay.?? jdk.internal.vm.compiler.management will become empty until the next Graal update and I guess this makes it easier to make Graal change in the upstream project.?? This is fine with me. Mandy From david.holmes at oracle.com Fri Apr 13 05:15:08 2018 From: david.holmes at oracle.com (David Holmes) Date: Fri, 13 Apr 2018 15:15:08 +1000 Subject: RFR: 8187490: HotSpotRuntimeMBean should be moved to Graal management module In-Reply-To: <324AC8E7-9737-4DB9-875E-A7FDB551D75E@oracle.com> References: <324AC8E7-9737-4DB9-875E-A7FDB551D75E@oracle.com> Message-ID: Hi Doug, Not a review. :) Just wondering what HotSpotRuntimeMBean has to do with this ??? Thanks, David On 13/04/2018 4:24 AM, Doug Simon wrote: > Please review this change that removes the existing Graal service provider for hooking into the Platform MBean Server and makes jdk.internal.vm.compiler.management an upgradeable module. > > Please refer to https://bugs.openjdk.java.net/browse/JDK-8187490?focusedCommentId=14170942&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14170942 for discussion on the latter point. > > The Graal changes that dynamically register an MBean for accessing Graal will be part of a subsequent Graal update. > > http://cr.openjdk.java.net/~dnsimon/8187490/ > > -Doug > From Alan.Bateman at oracle.com Fri Apr 13 06:27:25 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 13 Apr 2018 07:27:25 +0100 Subject: RFR: 8187490: HotSpotRuntimeMBean should be moved to Graal management module In-Reply-To: <324AC8E7-9737-4DB9-875E-A7FDB551D75E@oracle.com> References: <324AC8E7-9737-4DB9-875E-A7FDB551D75E@oracle.com> Message-ID: On 12/04/2018 19:24, Doug Simon wrote: > Please review this change that removes the existing Graal service provider for hooking into the Platform MBean Server and makes jdk.internal.vm.compiler.management an upgradeable module. > > Please refer to https://bugs.openjdk.java.net/browse/JDK-8187490?focusedCommentId=14170942&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14170942 for discussion on the latter point. > > The Graal changes that dynamically register an MBean for accessing Graal will be part of a subsequent Graal update. > > http://cr.openjdk.java.net/~dnsimon/8187490/ > This looks okay to me. -Alan From peter.levart at gmail.com Fri Apr 13 06:48:10 2018 From: peter.levart at gmail.com (Peter Levart) Date: Fri, 13 Apr 2018 08:48:10 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> <2b141a11-ebef-ca86-d37f-e9060d309e82@oracle.com> <7aee3cd3-dc2e-3fc6-5cb6-0192d3f17943@oracle.com> Message-ID: <134535b5-10da-f440-96a2-39f79b705137@gmail.com> On 04/12/18 10:40, Alan Bateman wrote: > On 11/04/2018 21:07, Rafael Winterhalter wrote: >> I do not think that this is possible. If the module containing the >> interface does not open a package, I cannot change the privileges of >> the main module such that I can resolve a method handle for invoking >> the special invocation. >> >> I just tried this out too and I did not find a way, could you suggest >> how to change my code for being able to do so? > If the interface is public in an exported package (no need for the > package to be open) then the proxy will be generated into the unnamed > module. So easy to get a Lookup to the proxy class and you can use > this as the special caller. Can you change your invocation handler to > the following and try it: > > Class proxyClass = proxy.getClass(); > Main.class.getModule().addReads(proxyClass.getModule()); > Lookup lookup = MethodHandles.privateLookupIn(proxyClass, > MethodHandles.lookup()); > MethodType mt = MethodType.methodType(String.class); > return lookup.findSpecial(iface, "foo", mt, > proxyClass).bindTo(proxy).invokeWithArguments(); > > -Alan This works logically, but performance wise I would 1st create a proxy class, lookup the direct method handles I want to invoke, transform them to take Object 'proxy' instance as 1st argument and the rest of arguments as an Object[] and return an Object. I would cache the resulting MHs in the specific (constant if possible) InvocationHandler. Each invocation would then only select the right cached MH as quickly as possible and do return mh.invokeExact(proxy, args); Peter From doug.simon at oracle.com Fri Apr 13 07:12:08 2018 From: doug.simon at oracle.com (Doug Simon) Date: Fri, 13 Apr 2018 09:12:08 +0200 Subject: RFR: 8187490: HotSpotRuntimeMBean should be moved to Graal management module In-Reply-To: References: <324AC8E7-9737-4DB9-875E-A7FDB551D75E@oracle.com> Message-ID: <9F2671A0-BD33-4081-8829-1E93CA622FDD@oracle.com> > On 13 Apr 2018, at 07:15, David Holmes wrote: > > Hi Doug, > > Not a review. :) Just wondering what HotSpotRuntimeMBean has to do with this ??? These are the non-Graal code base changes needed to move the bean out of the jdk.internal.vm.compiler module. The rest of the changes will come in the next Graal update. If you'd like, I can defer pushing these changes until the Graal changes land on github so that the complete change can be reviewed. -Doug > On 13/04/2018 4:24 AM, Doug Simon wrote: >> Please review this change that removes the existing Graal service provider for hooking into the Platform MBean Server and makes jdk.internal.vm.compiler.management an upgradeable module. >> Please refer to https://bugs.openjdk.java.net/browse/JDK-8187490?focusedCommentId=14170942&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14170942 for discussion on the latter point. >> The Graal changes that dynamically register an MBean for accessing Graal will be part of a subsequent Graal update. >> http://cr.openjdk.java.net/~dnsimon/8187490/ >> -Doug From doug.simon at oracle.com Fri Apr 13 12:16:27 2018 From: doug.simon at oracle.com (Doug Simon) Date: Fri, 13 Apr 2018 14:16:27 +0200 Subject: Provides clauses in binary module descriptor but not in source Message-ID: I just noticed that in the jdk.internal.vm.compiler module descriptor source there is a `uses` clause for CompilerConfigurationFactory[1] but no `provides` clause for the CoreCompilerConfigurationFactory provider[2] which is in the same module. However, `java -d jdk.internal.vm.compiler | grep Core` shows me the provider clause exists in the binary module descriptor. Is this done auto-magically by javac when building the module? If not, is it in the make files somewhere? I'm asking because there are new service providers being added in Graal. -Doug [1] http://hg.openjdk.java.net/jdk/jdk/file/2918e1146106/src/jdk.internal.vm.compiler/share/classes/module-info.java#l36 [2] http://hg.openjdk.java.net/jdk/jdk/file/2918e1146106/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot/src/org/graalvm/compiler/hotspot/CoreCompilerConfigurationFactory.java#l30 From Alan.Bateman at oracle.com Fri Apr 13 12:33:07 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 13 Apr 2018 13:33:07 +0100 Subject: Provides clauses in binary module descriptor but not in source In-Reply-To: References: Message-ID: <2237ade1-063c-45f0-6a6f-de7701b38a19@oracle.com> On 13/04/2018 13:16, Doug Simon wrote: > I just noticed that in the jdk.internal.vm.compiler module descriptor source there is a `uses` clause for CompilerConfigurationFactory[1] but no `provides` clause for the CoreCompilerConfigurationFactory provider[2] which is in the same module. However, `java -d jdk.internal.vm.compiler | grep Core` shows me the provider clause exists in the binary module descriptor. Is this done auto-magically by javac when building the module? If not, is it in the make files somewhere? I'm asking because there are new service providers being added in Graal. > The build for that module is complex as it runs an annotation processor and generates a module-info.java.extra (see support/gensrc/jdk.internal.vm.compiler/ in the build output) that is merged with the module-info.java before it is compiled. So no javac magic. -Alan From doug.simon at oracle.com Fri Apr 13 12:45:03 2018 From: doug.simon at oracle.com (Doug Simon) Date: Fri, 13 Apr 2018 14:45:03 +0200 Subject: Provides clauses in binary module descriptor but not in source In-Reply-To: <2237ade1-063c-45f0-6a6f-de7701b38a19@oracle.com> References: <2237ade1-063c-45f0-6a6f-de7701b38a19@oracle.com> Message-ID: <3DF822E4-DB6A-40A1-8B19-7D2FABBDA7FC@oracle.com> > On 13 Apr 2018, at 14:33, Alan Bateman wrote: > > On 13/04/2018 13:16, Doug Simon wrote: >> I just noticed that in the jdk.internal.vm.compiler module descriptor source there is a `uses` clause for CompilerConfigurationFactory[1] but no `provides` clause for the CoreCompilerConfigurationFactory provider[2] which is in the same module. However, `java -d jdk.internal.vm.compiler | grep Core` shows me the provider clause exists in the binary module descriptor. Is this done auto-magically by javac when building the module? If not, is it in the make files somewhere? I'm asking because there are new service providers being added in Graal. >> > The build for that module is complex as it runs an annotation processor and generates a module-info.java.extra (see support/gensrc/jdk.internal.vm.compiler/ in the build output) that is merged with the module-info.java before it is compiled. So no javac magic. Ah yes, I'd forgotten that the @ServiceProvider annotation was run my the JDK make system. Thanks for the helpful reminder; it means we can simply use this annotation for the new service providers we're adding and everything should Just Work. -Doug From david.holmes at oracle.com Fri Apr 13 13:59:22 2018 From: david.holmes at oracle.com (David Holmes) Date: Fri, 13 Apr 2018 23:59:22 +1000 Subject: RFR: 8187490: HotSpotRuntimeMBean should be moved to Graal management module In-Reply-To: <9F2671A0-BD33-4081-8829-1E93CA622FDD@oracle.com> References: <324AC8E7-9737-4DB9-875E-A7FDB551D75E@oracle.com> <9F2671A0-BD33-4081-8829-1E93CA622FDD@oracle.com> Message-ID: <1961d59c-e37c-f98d-5442-4d857e74faf5@oracle.com> On 13/04/2018 5:12 PM, Doug Simon wrote: > > >> On 13 Apr 2018, at 07:15, David Holmes wrote: >> >> Hi Doug, >> >> Not a review. :) Just wondering what HotSpotRuntimeMBean has to do with this ??? > > These are the non-Graal code base changes needed to move the bean out of the jdk.internal.vm.compiler module. The rest of the changes will come in the next Graal update. If you'd like, I can defer pushing these changes until the Graal changes land on github so that the complete change can be reviewed. So we seem to have both HotSpotRuntimeMBean and HotspotRuntimeMBean (note small 's') defined in the source code! That seems to be a bad thing to me! I was wondering what this had to do with the small 's' HotspotRuntimeMBean - and the answer seems to be "nothing"! David > -Doug > >> On 13/04/2018 4:24 AM, Doug Simon wrote: >>> Please review this change that removes the existing Graal service provider for hooking into the Platform MBean Server and makes jdk.internal.vm.compiler.management an upgradeable module. >>> Please refer to https://bugs.openjdk.java.net/browse/JDK-8187490?focusedCommentId=14170942&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14170942 for discussion on the latter point. >>> The Graal changes that dynamically register an MBean for accessing Graal will be part of a subsequent Graal update. >>> http://cr.openjdk.java.net/~dnsimon/8187490/ >>> -Doug > From doug.simon at oracle.com Fri Apr 13 14:05:52 2018 From: doug.simon at oracle.com (Doug Simon) Date: Fri, 13 Apr 2018 16:05:52 +0200 Subject: RFR: 8187490: HotSpotRuntimeMBean should be moved to Graal management module In-Reply-To: <1961d59c-e37c-f98d-5442-4d857e74faf5@oracle.com> References: <324AC8E7-9737-4DB9-875E-A7FDB551D75E@oracle.com> <9F2671A0-BD33-4081-8829-1E93CA622FDD@oracle.com> <1961d59c-e37c-f98d-5442-4d857e74faf5@oracle.com> Message-ID: > On 13 Apr 2018, at 15:59, David Holmes wrote: > > On 13/04/2018 5:12 PM, Doug Simon wrote: >>> On 13 Apr 2018, at 07:15, David Holmes wrote: >>> >>> Hi Doug, >>> >>> Not a review. :) Just wondering what HotSpotRuntimeMBean has to do with this ??? >> These are the non-Graal code base changes needed to move the bean out of the jdk.internal.vm.compiler module. The rest of the changes will come in the next Graal update. If you'd like, I can defer pushing these changes until the Graal changes land on github so that the complete change can be reviewed. > > So we seem to have both HotSpotRuntimeMBean and HotspotRuntimeMBean (note small 's') defined in the source code! That seems to be a bad thing to me! I was wondering what this had to do with the small 's' HotspotRuntimeMBean - and the answer seems to be "nothing"! There is actually no HotSpotRuntimeMBean.java source - it's HotSpotGraalMBean.java. I've renamed the issue and fixed the naming in the description. -Doug >> -Doug >>> On 13/04/2018 4:24 AM, Doug Simon wrote: >>>> Please review this change that removes the existing Graal service provider for hooking into the Platform MBean Server and makes jdk.internal.vm.compiler.management an upgradeable module. >>>> Please refer to https://bugs.openjdk.java.net/browse/JDK-8187490?focusedCommentId=14170942&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14170942 for discussion on the latter point. >>>> The Graal changes that dynamically register an MBean for accessing Graal will be part of a subsequent Graal update. >>>> http://cr.openjdk.java.net/~dnsimon/8187490/ >>>> -Doug From rafael.wth at gmail.com Fri Apr 13 19:49:54 2018 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Fri, 13 Apr 2018 21:49:54 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: <134535b5-10da-f440-96a2-39f79b705137@gmail.com> References: <808e7629-3da5-3931-982c-94da4e29be71@oracle.com> <2b141a11-ebef-ca86-d37f-e9060d309e82@oracle.com> <7aee3cd3-dc2e-3fc6-5cb6-0192d3f17943@oracle.com> <134535b5-10da-f440-96a2-39f79b705137@gmail.com> Message-ID: Yes, of course. This was merely a proof of concept to show that it works. 2018-04-13 8:48 GMT+02:00 Peter Levart : > > > On 04/12/18 10:40, Alan Bateman wrote: > > On 11/04/2018 21:07, Rafael Winterhalter wrote: > > I do not think that this is possible. If the module containing the > interface does not open a package, I cannot change the privileges of the > main module such that I can resolve a method handle for invoking the > special invocation. > > I just tried this out too and I did not find a way, could you suggest how > to change my code for being able to do so? > > If the interface is public in an exported package (no need for the package > to be open) then the proxy will be generated into the unnamed module. So > easy to get a Lookup to the proxy class and you can use this as the special > caller. Can you change your invocation handler to the following and try it: > > Class proxyClass = proxy.getClass(); > Main.class.getModule().addReads(proxyClass.getModule()); > Lookup lookup = MethodHandles.privateLookupIn(proxyClass, > MethodHandles.lookup()); > MethodType mt = MethodType.methodType(String.class); > return lookup.findSpecial(iface, "foo", mt, proxyClass).bindTo(proxy).invokeWithArguments(); > > > -Alan > > > This works logically, but performance wise I would 1st create a proxy > class, lookup the direct method handles I want to invoke, transform them to > take Object 'proxy' instance as 1st argument and the rest of arguments as > an Object[] and return an Object. I would cache the resulting MHs in the > specific (constant if possible) InvocationHandler. Each invocation would > then only select the right cached MH as quickly as possible and do return > mh.invokeExact(proxy, args); > > Peter > > From mandy.chung at oracle.com Sun Apr 15 06:33:13 2018 From: mandy.chung at oracle.com (mandy chung) Date: Sun, 15 Apr 2018 14:33:13 +0800 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: References: Message-ID: <34e4b4e5-6716-ef7e-b2a4-0125c19de616@oracle.com> Hi Rafael, FYI.? I have sent a proposal [1] for item #1 to serviceability-dev and you for discussion.? JBS issue is: ?? https://bugs.openjdk.java.net/browse/JDK-8200559 Mandy [1] http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-April/023529.html On 4/2/18 5:02 AM, Rafael Winterhalter wrote: > 1. Java agents cannot define auxiliary classes. > > Byte Buddy does support the JPMS fully, however, it still relies on > sun.misc.Unsafe::defineClass for its Java agent API and currently breaks on > Java 11 as this method was removed in a recent EA build. The reason for > using Unsafe is that many instrumentations need to define auxiliary classes > to aid an instrumentation similar to javac which sometimes needs to define > anonymous classes or even synthetic classes. For example, if a Java agent > wants to register an event listener to some framework, such listeners often > declare multiple methods what makes it impossible to fullfil the listener > contract using a lambda expression. Instead, one typically injects an > additional class into the same package as the instrumented class. In this > case, it is not possible to use MethodHandles.Lookup::defineClass as the > class file transformer does not necessarily have private access to the > lookup of the instrumented class. > > The current workarounds are: > > a) Open the package jdk.internal.misc to gain access to this package's > Unsafe class. This can be done via Instrumentation::redefineModule. > b) Open the java.lang package to access ClassLoader via reflection. > c) Open the java.lang package to access the internal lookup with global > access rights. > > Of these solutions only (b) relies on standard API and is guaranteed to > function in the future but the solution still feels hacky and does not work > for instrumentations of classes on the bootstrap loader. Opening packages > also implies a risk of being applied carelessly since opening the package > to the agent's module most likely opens the package to the unnamed module > of the system class loader what invites to breaches of the JPMS > encapsulation by code that does not ship with the agent. > To offer a better solution, I would like to suggest one of the following: > > a) Add a method defineClass(ClassLoader, byte[], ProtectionDomain) to the > Instrumentation interface that works similar to Unsafe::defineClass. This > would provide a very simple migration path. Since agents have access to > jdk.internal.misc, adding this method does not add any capabilities to the > agent, it merley avoids using internal API that might change. > b) Supply a MethodHandles.Lookup instance to the > ClassFileTransformer::transform API where the instance represents the > instrumented class's access rights. This does however carry the risk of > invoking the lookupClass method which would either load the instrumented > class prematurely causing a circularity error or return an unexpected value > such as null. Since the lookup API generally relies on loaded types, there > are probably other problems such as invoking Lookup::privateLookupIn before > all involved types are loaded. > > For the sake of simlicity and since easy migration paths make a quick > adoption easier, I would suggestion solution (a), also in the light that > quick and dirty migrations might still choose option (b) to save time and > also since (b) might cause problems when types are not yet loaded. > From rafael.wth at gmail.com Sun Apr 15 20:43:15 2018 From: rafael.wth at gmail.com (Rafael Winterhalter) Date: Sun, 15 Apr 2018 22:43:15 +0200 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: <34e4b4e5-6716-ef7e-b2a4-0125c19de616@oracle.com> References: <34e4b4e5-6716-ef7e-b2a4-0125c19de616@oracle.com> Message-ID: Thanks Mandy, I appreciate it! Best regards, Rafael 2018-04-15 8:33 GMT+02:00 mandy chung : > Hi Rafael, > > FYI. I have sent a proposal [1] for item #1 to serviceability-dev and you > for discussion. JBS issue is: > https://bugs.openjdk.java.net/browse/JDK-8200559 > > Mandy > [1] http://mail.openjdk.java.net/pipermail/serviceability-dev/ > 2018-April/023529.html > > > On 4/2/18 5:02 AM, Rafael Winterhalter wrote: > > 1. Java agents cannot define auxiliary classes. > > Byte Buddy does support the JPMS fully, however, it still relies on > sun.misc.Unsafe::defineClass for its Java agent API and currently breaks on > Java 11 as this method was removed in a recent EA build. The reason for > using Unsafe is that many instrumentations need to define auxiliary classes > to aid an instrumentation similar to javac which sometimes needs to define > anonymous classes or even synthetic classes. For example, if a Java agent > wants to register an event listener to some framework, such listeners often > declare multiple methods what makes it impossible to fullfil the listener > contract using a lambda expression. Instead, one typically injects an > additional class into the same package as the instrumented class. In this > case, it is not possible to use MethodHandles.Lookup::defineClass as the > class file transformer does not necessarily have private access to the > lookup of the instrumented class. > > The current workarounds are: > > a) Open the package jdk.internal.misc to gain access to this package's > Unsafe class. This can be done via Instrumentation::redefineModule. > b) Open the java.lang package to access ClassLoader via reflection. > c) Open the java.lang package to access the internal lookup with global > access rights. > > Of these solutions only (b) relies on standard API and is guaranteed to > function in the future but the solution still feels hacky and does not work > for instrumentations of classes on the bootstrap loader. Opening packages > also implies a risk of being applied carelessly since opening the package > to the agent's module most likely opens the package to the unnamed module > of the system class loader what invites to breaches of the JPMS > encapsulation by code that does not ship with the agent. > > To offer a better solution, I would like to suggest one of the following: > > a) Add a method defineClass(ClassLoader, byte[], ProtectionDomain) to the > Instrumentation interface that works similar to Unsafe::defineClass. This > would provide a very simple migration path. Since agents have access to > jdk.internal.misc, adding this method does not add any capabilities to the > agent, it merley avoids using internal API that might change. > b) Supply a MethodHandles.Lookup instance to the > ClassFileTransformer::transform API where the instance represents the > instrumented class's access rights. This does however carry the risk of > invoking the lookupClass method which would either load the instrumented > class prematurely causing a circularity error or return an unexpected value > such as null. Since the lookup API generally relies on loaded types, there > are probably other problems such as invoking Lookup::privateLookupIn before > all involved types are loaded. > > For the sake of simlicity and since easy migration paths make a quick > adoption easier, I would suggestion solution (a), also in the light that > quick and dirty migrations might still choose option (b) to save time and > also since (b) might cause problems when types are not yet loaded. > > > From adinn at redhat.com Mon Apr 16 09:29:37 2018 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 16 Apr 2018 10:29:37 +0100 Subject: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links In-Reply-To: <34e4b4e5-6716-ef7e-b2a4-0125c19de616@oracle.com> References: <34e4b4e5-6716-ef7e-b2a4-0125c19de616@oracle.com> Message-ID: <1c16493a-4153-1946-5c40-328c33ad6df5@redhat.com> On 15/04/18 07:33, mandy chung wrote: > FYI.? I have sent a proposal [1] for item #1 to serviceability-dev and > you for discussion.? JBS issue is: > ?? https://bugs.openjdk.java.net/browse/JDK-8200559 > > Mandy > [1] > http://mail.openjdk.java.net/pipermail/serviceability-dev/2018-April/023529.html Thanks, Mandy. The proposed API looks great. Good luck with those corner-cases in the implementation :-) regards, Andrew Dinn ----------- From doug.simon at oracle.com Wed Apr 18 10:35:22 2018 From: doug.simon at oracle.com (Doug Simon) Date: Wed, 18 Apr 2018 12:35:22 +0200 Subject: RFR: 8201794: [Graal] add dummy HotSpotGraalManagement class Message-ID: Right after pushing the change for JDK-8187490, I noticed that the mach5 build had 2 CTW test failures due to jdk.internal.vm.compiler.management now being an empty module. This will be fixed in the next Graal update but the failing tests should be fixed in the meantime. http://cr.openjdk.java.net/~dnsimon/8201794/ https://bugs.openjdk.java.net/browse/JDK-8201794 I'm running mach5 now to ensure the regressions are fixed and no other regressions occur. -Doug From doug.simon at oracle.com Wed Apr 18 11:00:52 2018 From: doug.simon at oracle.com (Doug Simon) Date: Wed, 18 Apr 2018 13:00:52 +0200 Subject: RFR: 8201794: [Graal] fix regressions from JDK-8187490 In-Reply-To: References: Message-ID: <779B4D53-750C-46C5-A907-05B54B53BBCC@oracle.com> I've updated the webrev with a fix for another regression caused by JDK-8187490: --- old/test/jdk/jdk/modules/etc/UpgradeableModules.java 2018-04-18 12:57:32.000000000 +0200 +++ new/test/jdk/jdk/modules/etc/UpgradeableModules.java 2018-04-18 12:57:32.000000000 +0200 @@ -46,6 +46,7 @@ List.of("java.compiler", "java.jnlp", "jdk.internal.vm.compiler", + "jdk.internal.vm.compiler.management", "jdk.deploy", "jdk.javaws", "jdk.plugin", -Doug > On 18 Apr 2018, at 12:35, Doug Simon wrote: > > Right after pushing the change for JDK-8187490, I noticed that the mach5 build had 2 CTW test failures due to jdk.internal.vm.compiler.management now being an empty module. This will be fixed in the next Graal update but the failing tests should be fixed in the meantime. > > http://cr.openjdk.java.net/~dnsimon/8201794/ > https://bugs.openjdk.java.net/browse/JDK-8201794 > > I'm running mach5 now to ensure the regressions are fixed and no other regressions occur. > > -Doug From mandy.chung at oracle.com Wed Apr 18 11:23:23 2018 From: mandy.chung at oracle.com (mandy chung) Date: Wed, 18 Apr 2018 19:23:23 +0800 Subject: RFR: 8201794: [Graal] fix regressions from JDK-8187490 In-Reply-To: <779B4D53-750C-46C5-A907-05B54B53BBCC@oracle.com> References: <779B4D53-750C-46C5-A907-05B54B53BBCC@oracle.com> Message-ID: <2b4c66f4-bf98-83e2-0ebf-690fb70d7bf2@oracle.com> Looks okay.? Have you run all jdk_core tests in addition to other hotspot tests. Mandy On 4/18/18 7:00 PM, Doug Simon wrote: > I've updated the webrev with a fix for another regression caused by JDK-8187490: > > --- old/test/jdk/jdk/modules/etc/UpgradeableModules.java 2018-04-18 12:57:32.000000000 +0200 > +++ new/test/jdk/jdk/modules/etc/UpgradeableModules.java 2018-04-18 12:57:32.000000000 +0200 > @@ -46,6 +46,7 @@ > List.of("java.compiler", > "java.jnlp", > "jdk.internal.vm.compiler", > + "jdk.internal.vm.compiler.management", > "jdk.deploy", > "jdk.javaws", > "jdk.plugin", > > -Doug > >> On 18 Apr 2018, at 12:35, Doug Simon wrote: >> >> Right after pushing the change for JDK-8187490, I noticed that the mach5 build had 2 CTW test failures due to jdk.internal.vm.compiler.management now being an empty module. This will be fixed in the next Graal update but the failing tests should be fixed in the meantime. >> >> http://cr.openjdk.java.net/~dnsimon/8201794/ >> https://bugs.openjdk.java.net/browse/JDK-8201794 >> >> I'm running mach5 now to ensure the regressions are fixed and no other regressions occur. >> >> -Doug From doug.simon at oracle.com Wed Apr 18 11:28:21 2018 From: doug.simon at oracle.com (Doug Simon) Date: Wed, 18 Apr 2018 13:28:21 +0200 Subject: RFR: 8201794: [Graal] fix regressions from JDK-8187490 In-Reply-To: <2b4c66f4-bf98-83e2-0ebf-690fb70d7bf2@oracle.com> References: <779B4D53-750C-46C5-A907-05B54B53BBCC@oracle.com> <2b4c66f4-bf98-83e2-0ebf-690fb70d7bf2@oracle.com> Message-ID: > On 18 Apr 2018, at 13:23, mandy chung wrote: > > Looks okay. Have you run all jdk_core tests in addition to other hotspot tests. I ran all open/test/jdk/jdk tests - is that what you mean? -Doug > On 4/18/18 7:00 PM, Doug Simon wrote: >> I've updated the webrev with a fix for another regression caused by JDK-8187490: >> >> --- old/test/jdk/jdk/modules/etc/UpgradeableModules.java 2018-04-18 12:57:32.000000000 +0200 >> +++ new/test/jdk/jdk/modules/etc/UpgradeableModules.java 2018-04-18 12:57:32.000000000 +0200 >> @@ -46,6 +46,7 @@ >> List.of("java.compiler", >> "java.jnlp", >> "jdk.internal.vm.compiler", >> + "jdk.internal.vm.compiler.management", >> "jdk.deploy", >> "jdk.javaws", >> "jdk.plugin", >> >> -Doug >> >> >>> On 18 Apr 2018, at 12:35, Doug Simon >>> wrote: >>> >>> Right after pushing the change for JDK-8187490, I noticed that the mach5 build had 2 CTW test failures due to jdk.internal.vm.compiler.management now being an empty module. This will be fixed in the next Graal update but the failing tests should be fixed in the meantime. >>> >>> >>> http://cr.openjdk.java.net/~dnsimon/8201794/ >>> https://bugs.openjdk.java.net/browse/JDK-8201794 >>> >>> >>> I'm running mach5 now to ensure the regressions are fixed and no other regressions occur. >>> >>> -Doug >>> > From Alan.Bateman at oracle.com Wed Apr 18 11:28:29 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 18 Apr 2018 12:28:29 +0100 Subject: RFR: 8201794: [Graal] fix regressions from JDK-8187490 In-Reply-To: <779B4D53-750C-46C5-A907-05B54B53BBCC@oracle.com> References: <779B4D53-750C-46C5-A907-05B54B53BBCC@oracle.com> Message-ID: On 18/04/2018 12:00, Doug Simon wrote: > I've updated the webrev with a fix for another regression caused by JDK-8187490: > > --- old/test/jdk/jdk/modules/etc/UpgradeableModules.java 2018-04-18 12:57:32.000000000 +0200 > +++ new/test/jdk/jdk/modules/etc/UpgradeableModules.java 2018-04-18 12:57:32.000000000 +0200 > @@ -46,6 +46,7 @@ > List.of("java.compiler", > "java.jnlp", > "jdk.internal.vm.compiler", > + "jdk.internal.vm.compiler.management", > "jdk.deploy", > "jdk.javaws", > "jdk.plugin", > This looks fine, this seems to be the only jdk_core test failing after JDK-8187490. -Alan From mike at plan99.net Mon Apr 23 17:21:33 2018 From: mike at plan99.net (Mike Hearn) Date: Mon, 23 Apr 2018 10:21:33 -0700 Subject: JMOD, native libraries and the packaging of JavaFX Message-ID: Hello, The JMOD format is not documented directly, but is essentially a JAR-like format which can also contain native libraries, license texts, man pages and config files. However, JMODs are not just JARs with extra features. They are incompatible, because class files go under a classes/ directory. JEP 261 says: JMOD files can be used at compile time and link time, but not at run time. To support them at run time would require, in general, that we be prepared to extract and link native-code libraries on-the-fly. This is feasible on most platforms, though it can be very tricky, and we have not seen many use cases that require this capability, so for simplicity we have chosen to limit the utility of JMOD files in this release. I was a bit surprised when I first read this because JARs that contain native libraries, along with hacky custom code to extract and load them, are actually quite common. One example is Conscrypt but there are others. An extended JAR format in which the JVM took care of loading the right native library would be a very helpful thing to have. Whilst the task can be tricky if you want to do it as efficiently as possible e.g. not save the DLL / DSO to disk, a "good enough" approach can't be that hard because the community has implemented it many times. I'm thinking about this issue now because I quite like JavaFX and its future is clearly as a regular Java library, albeit a big one, distributed either via (not ideal) an SDK or (better) a set of regular libraries published to a Maven repository. Publishing JavaFX as a set of modules that developers can depend on with Maven or Gradle will require either that JavaFX include the sort of hacky extract-to-a-temp-dir-and-load code that is standard today, or that it's not published as ordinary JARs at all, or that the JPMS is extended in time for Java 11 to provide a uniform solution. As far as I can see the JMOD format is probably not going to gain adoption. Whilst it can be used by developers in theory: - There's no support in Maven or Gradle. - The format isn't documented. - Using it requires jlinking, which isn't a part of the regular developer workflow. - jlink doesn't do much for the vast majority of apps that can't be fully modularised yet. - It's not clear why it's better than an extended JAR format. - It behaves in puzzling ways, for example the "jar" tool can print the file listing of a jmod but not extract it. The bulk of the JMOD feature set could be delivered with two small extensions to the JAR format: 1. A common directory structure for storing native libraries. That would allow native library extraction and loading to be provided via a small library, if not provided by the JVM itself. 2. A common directory structure for including license files (this just has to be announced, as nothing needs to load them). 3. Metadata linking command line program names to main class names and JVM parameters. For example the jdk.javadoc.jmod contains a program bin/javadoc which is a native binary specific to the host platform of the JDK. But all it does is run the JavaDoc program itself, which is written in Java. This sort of startup program could be easily generated on the fly given startup parameters in the same way the javapackager tool does it. If the JAR format was extended in this way, it would become possible to write a tool that given a Maven coordinate would resolve and then install into a bin/ directory on the path a starter program with the suggested name. This would be a nice feature to have (I am writing such a tool already, but it'd be good to standardise the way to express a desired 'short name'). I realise there isn't much time left until the Java 11 ship sails. But for JavaFX users at least, it'd be convenient to have this small extension. Even if the JVM isn't extended to support it in 11, a common format would allow it to take over responsibility later. thanks, -mike From greggwon at cox.net Mon Apr 23 23:25:52 2018 From: greggwon at cox.net (Gregg Wonderly) Date: Mon, 23 Apr 2018 18:25:52 -0500 Subject: JMOD, native libraries and the packaging of JavaFX In-Reply-To: References: Message-ID: <7D379FD1-855B-4706-8F4F-F6A79CCFFEA6@cox.net> Yes, I, like many others have carried around various JNI code in jar files (javax.comm and my own as well), and then copied these out of the jar, into ?temp? space, and then used load() to load and bind them in class wrappers. This works quite well, but it is highly customized in how it?s implemented, but the effect is the same. Having the module system support modules with JNI resources would be extremely beneficial given the staleness and complete lack of work on bring more native functionality into the JVM to support things like JavaFX and even JMF which has died on the vine due to the complexity of users tasks to get JMF for use of such based applications. It?s clear that the desktop and non-web server JEE or such servers is not interesting to Oracle, but it?s quite actively pursued on the platform and ignoring that fact will just continue to erode the use of Java for portable solutions that it once was trumpeted at being the key to. Gregg > On Apr 23, 2018, at 12:21 PM, Mike Hearn wrote: > > Hello, > > The JMOD format is not documented directly, but is essentially a JAR-like > format which can also contain native libraries, license texts, man pages > and config files. However, JMODs are not just JARs with extra features. > They are incompatible, because class files go under a classes/ directory. > > JEP 261 says: > > JMOD files can be used at compile time and link time, but not at run time. > To support them at run time would require, in general, that we be prepared > to extract and link native-code libraries on-the-fly. This is feasible on > most platforms, though it can be very tricky, and we have not seen many use > cases that require this capability, so for simplicity we have chosen to > limit the utility of JMOD files in this release. > > > I was a bit surprised when I first read this because JARs that contain > native libraries, along with hacky custom code to extract and load them, > are actually quite common. One example is Conscrypt but there are others. > An extended JAR format in which the JVM took care of loading the right > native library would be a very helpful thing to have. Whilst the task can > be tricky if you want to do it as efficiently as possible e.g. not save the > DLL / DSO to disk, a "good enough" approach can't be that hard because the > community has implemented it many times. > > I'm thinking about this issue now because I quite like JavaFX and its > future is clearly as a regular Java library, albeit a big one, distributed > either via (not ideal) an SDK or (better) a set of regular libraries > published to a Maven repository. > > Publishing JavaFX as a set of modules that developers can depend on with > Maven or Gradle will require either that JavaFX include the sort of hacky > extract-to-a-temp-dir-and-load code that is standard today, or that it's > not published as ordinary JARs at all, or that the JPMS is extended in time > for Java 11 to provide a uniform solution. > > As far as I can see the JMOD format is probably not going to gain adoption. > Whilst it can be used by developers in theory: > > - There's no support in Maven or Gradle. > - The format isn't documented. > - Using it requires jlinking, which isn't a part of the regular > developer workflow. > - jlink doesn't do much for the vast majority of apps that can't be > fully modularised yet. > - It's not clear why it's better than an extended JAR format. > - It behaves in puzzling ways, for example the "jar" tool can print the > file listing of a jmod but not extract it. > > The bulk of the JMOD feature set could be delivered with two small > extensions to the JAR format: > > 1. A common directory structure for storing native libraries. That would > allow native library extraction and loading to be provided via a small > library, if not provided by the JVM itself. > 2. A common directory structure for including license files (this just > has to be announced, as nothing needs to load them). > 3. Metadata linking command line program names to main class names and > JVM parameters. > > For example the jdk.javadoc.jmod contains a program bin/javadoc which is a > native binary specific to the host platform of the JDK. But all it does is > run the JavaDoc program itself, which is written in Java. This sort of > startup program could be easily generated on the fly given startup > parameters in the same way the javapackager tool does it. > > If the JAR format was extended in this way, it would become possible to > write a tool that given a Maven coordinate would resolve and then install > into a bin/ directory on the path a starter program with the suggested > name. This would be a nice feature to have (I am writing such a tool > already, but it'd be good to standardise the way to express a desired > 'short name'). > > I realise there isn't much time left until the Java 11 ship sails. But for > JavaFX users at least, it'd be convenient to have this small extension. > Even if the JVM isn't extended to support it in 11, a common format would > allow it to take over responsibility later. > > thanks, > -mike From stephan.herrmann at berlin.de Tue Apr 24 18:55:34 2018 From: stephan.herrmann at berlin.de (Stephan Herrmann) Date: Tue, 24 Apr 2018 20:55:34 +0200 Subject: Deprecation beyond the level of types In-Reply-To: <5AC2CFAA.3050704@oracle.com> References: <5AC2CFAA.3050704@oracle.com> Message-ID: <01e7b336-6aeb-21d7-bcb9-7e2ac8fb5b75@berlin.de> Thanks, Alex, for your answer, and sorry for the long delay on my side. I found the background regarding deprecation of packages quite interesting, and agree to most of what you say. On 03.04.2018 02:49, Alex Buckley wrote: > You're right -- the JLS does not mandate a warning when compiling code on the classpath that uses code in deprecated modules. Wouldn't it be more consistent to give such warnings? Why should writing code in an unnamed vs. named module make a difference in whether or not I'm seeing deprecation warnings? Would you consider a compiler that issues such warnings as violating JLS? thanks, Stephan From alex.buckley at oracle.com Tue Apr 24 19:32:28 2018 From: alex.buckley at oracle.com (Alex Buckley) Date: Tue, 24 Apr 2018 12:32:28 -0700 Subject: Deprecation beyond the level of types In-Reply-To: <01e7b336-6aeb-21d7-bcb9-7e2ac8fb5b75@berlin.de> References: <5AC2CFAA.3050704@oracle.com> <01e7b336-6aeb-21d7-bcb9-7e2ac8fb5b75@berlin.de> Message-ID: <5ADF864C.503@oracle.com> On 4/24/2018 11:55 AM, Stephan Herrmann wrote: > On 03.04.2018 02:49, Alex Buckley wrote: >> You're right -- the JLS does not mandate a warning when compiling code >> on the classpath that uses code in deprecated modules. > > Wouldn't it be more consistent to give such warnings? > Why should writing code in an unnamed vs. named module make a > difference in whether or not I'm seeing deprecation warnings? Consistent yes, usable no. The possibility of a module being deprecated is not understood by vast amounts of code on the classpath written before Java SE 9. If deprecating a module implicitly deprecated all its exported public types, then code on the classpath would start to see warnings "out of thin air" -- nothing that the classpath code refers to by name would be explicitly deprecated. We thought such "action at a distance" was too great a burden to impose on classpath code. (And we are consistent in not imposing "action at a distance" -- applying @SuppressWarnings to a module declaration does NOT implicitly suppress warnings for all the type declarations in the module.) > Would you consider a compiler that issues such warnings as violating JLS? A compiler can only issue _deprecation warnings_ (and their brother, _removal warnings_) where the JLS mandates it. But a compiler is free to issue other kinds of warnings anywhere. For example, javac has -Xlint:dep-ann which issues warnings related to, but not exactly about, deprecation. Alex From everson.c.junior at gmail.com Mon Apr 30 20:57:08 2018 From: everson.c.junior at gmail.com (Everson Ferreira da Cunha Junior) Date: Mon, 30 Apr 2018 20:57:08 +0000 Subject: Will Java be paid Message-ID: People, Will Java be paid next years ?