From neil.bartlett at paremus.com Tue Nov 1 21:17:43 2016 From: neil.bartlett at paremus.com (Neil Bartlett (Paremus)) Date: Tue, 1 Nov 2016 21:17:43 +0000 Subject: Proposal (revised^2): #ReflectiveAccessToNonExportedTypes & #AwkwardStrongEncapsulation: Open modules & open packages In-Reply-To: <20161027155336.DBCE7DA329@eggemoggin.niobe.net> References: <20161027155336.DBCE7DA329@eggemoggin.niobe.net> Message-ID: <1C97C371-66E1-4770-9307-FEE6CB912B81@paremus.com> Dear experts, First thank you Mark for admitting me to this EG, and thank you to the other members for their kind welcome messages. With regards to this latest proposal for #ReflectiveAccessToNonExportedTypes, it is certainly better than the previous proposal (weak modules). However I find myself wondering whether ?open? should be the default? I anticipate use-cases in which a module author cannot predict whether or how the module will be accessed reflectively ? in this case she would not have used the open modifier and reflection will not be possible. In fact I believe that most module authors will open everything as a shortcut to compatibility with existing frameworks. For example, DI frameworks such as Spring or CDI perform classpath scanning to find annotated bean types. Web containers scan for annotated Servlets or JAX-RS Resource classes. It may be hard for developers using these framework to find the minimal set of packages or modules to open. There may be a small number of utility packages that can be closed, but wouldn't it be easier to explicitly close just those? By analogy with other levels in the Java language: classes are not final by default, and packages are not sealed by default. Finally, in the notes section of your proposal you recognise that it will be common to have open modules. However in the main text there is still some of the language about open modules being a migration step towards ?normal? modules, which is reminiscent of your earlier characterisation of weak modules. My concern with weak modules was that they could never be fully eliminated, so it was wrong to talk about them as purely transitional. Do you agree that there are valid use-cases in which even a well-modularised application contains a number of open modules? Regards, Neil > On 27 Oct 2016, at 16:53, mark.reinhold at oracle.com wrote: > > Issue summary > ------------- > > #ReflectiveAccessToNonExportedTypes --- Some kinds of framework > libraries require reflective access to members of the non-exported > types of other modules; examples include dependency injection (Guice), > persistence (JPA), debugging tools, code-automation tools, and > serialization (XStream). In some cases the particular library to be > used is not known until run time (e.g., Hibernate and EclipseLink both > implement JPA). This capability is also sometimes used to work around > bugs in unchangeable code. Access to non-exported packages can, at > present, only be done via command-line flags, which is extremely > awkward. Provide an easier way for reflective code to access such > non-exported types. [1] > > #AwkwardStrongEncapsulation --- A non-public element of an exported > package can still be accessed via the `AccessibleObject::setAccessible` > method of the core reflection API. The only way to strongly > encapsulate such an element is to move it to a non-exported package. > This makes it awkward, at best, to encapsulate the internals of a > package that defines a public API. [2] > > Proposal > -------- > > (This is the third proposal for #ReflectiveAccessToNonExportedTypes. > The note that follows [3] summarizes the background and relates the > alternatives presented thus far.) > > Extend the language of module declarations with the concept of _open_ > modules. Open modules make it easy to define modules whose internals > will be accessed at run time by reflection-based frameworks. Every > package in an open module has the following properties, by default: > > (1) At compile time it is not exported. > > (2) At run time it is exported without regard to split-package > conflicts or other inconsistencies, i.e., it does not affect > the operation of the resolver, nor the construction of layer > configurations, but is simply exported as if by invoking the > `Module::addExports` method during layer instantiation. > > (3) At run time it is available for _deep reflection_, i.e., all > of its elements are accessible via the core reflection API > including non-public elements, which can be accessed via the > `AccessibleObject::setAccessible` method. > > A package in an open module can be exported explicitly, for use at > compile time, via the familiar `exports` directive, so property (1) will > not hold. In that case it will also partake in run-time resolution, so > property (2) will not hold either. > > Suppose we have a module `foo.bar` that contains an API package > `com.foo.bar`, whose public types are intended for use by other modules, > and a non-API package `com.foo.bar.model`, that contains entity classes > to be manipulated by Hibernate via core reflection. Then the module > declaration > > open module foo.bar { > exports com.foo.bar; > requires hibernate.core; > requires hibernate.entitymanager; > } > > makes all elements, public and otherwise, of all packages available for > deep reflection at run time, but makes only the public and protected > types in `com.foo.bar` accessible at compile time. Thus Hibernate can > access non-public elements of the `com.foo.bar.model` package via the > `setAccessible` method, and some other module that `requires foo.bar` > can be compiled against the API in the `com.foo.bar` package but not > against the types in any other package. > > Open modules can be considered an intermediate step on the migration > path, between automatic modules and normal modules: > > - An automatic module offers the traditional level of encapsulation: > All packages are both open for deep reflective access and exported > for ordinary compile-time and run-time access to their public types. > > - An open module offers a modest degree of stronger encapsulation: All > packages are open for deep reflective access, but the module's author > must specify which packages, if any, are exported for ordinary > compile-time and run-time access. > > - A normal module offers the strongest encapsulation: The module's > author must specify which packages, if any, are open, or exported, > or both. > > An open module is a good starting point for application code that's being > modularized by its author. It affords a separation of concerns, so that > the author can focus on defining the module's domain-specific API without > having to worry about how the module will be inspected or manipulated at > run time by reflective frameworks. > > * * * > > To open specific packages in normal module declarations we introduce a > new per-package directive, `opens`. A package in a normal module can be > opened, or exported, or both, as follows: > > - If a package is only opened (`opens p;`) then properties (1), (2), > and (3) hold, i.e., its types are not accessible at compile time, > it does not affect resolution, and all of its elements are available > for deep reflection at run time. > > - If a package is only exported (`exports p;`) then those properties > do not hold, and it is exported exactly as it is today. > > - If a package is both exported and opened (`exports p; opens p;`) > then it is exported as today and is, additionally, available for > deep reflection at run time (3). > > We can use the `opens` directive to refine the previous example so that > only the `com.foo.bar` package is accessible at compile time, only the > `com.foo.bar.model` package is available for deep reflection at run time, > and the non-public elements of all other packages are strongly > encapsulated: > > module foo.bar { > exports com.foo.bar; > opens com.foo.bar.model; > requires hibernate.core; > requires hibernate.entitymanager; > } > > It's possible to both open and export a package in a normal module, but > it's often inadvisable. This is especially so for API packages, since > normally an API's internal implementation details should be strongly > encapsulated. This combination of directives may, however, be useful for > legacy APIs whose internals are known to be accessed by existing code. > > To ensure the integrity of the platform we expect that all the modules of > the JDK itself will be normal modules and that very few, if any, packages > will be opened. > > The `opens` directive cannot be used in an `open` module, since all > packages in such modules are implicitly open. > > * * * > > Both the `exports` and `opens` directives can be qualified, so that a > package is exported or opened only to certain other named modules. As > before, duplicate directives are not permitted in order to ensure easy > readability. At most one `exports` directive is relevant to any given > package, and at most one `opens` directive is relevant to any given > package. > > There is no syntax for wildcards. If the defaults are not sufficient for > a package then the package must be named explicitly. > > * * * > > The existing syntax of `requires public` has long been confusing, so we > here take the opportunity to fix that problem by renaming the `public` > modifier in `requires` directives to `transitive`. Thus the declaration > > module foo.bar { > exports com.foo.bar; > requires public java.sql; > } > > is now written > > module foo.bar { > exports com.foo.bar; > requires transitive java.sql; > } > > This is potentially confusing in a different way, since in mathematics > the term "transitive" is usually applied to an entire relation rather > than to three specific elements of a set. Its use here does not, in > particular, mean that the resolver does not interpret plain `requires` > directives when computing the transitive closure of a set of root > modules. "Transitive" as used here is in the more abstract sense, > expressing the notion of conveying a property -- in this case, the > readability of the required module -- from one thing to another. On > balance, however, `transitive` does appear superior to `public`. > > Notes > ----- > > - This proposal is significantly different from both the first and > second proposals [4][5]. It recognizes that, in practice, it will > be common to have open modules, all of whose content is available > for deep reflection but otherwise inaccessible at compile time unless > explicitly exported. If finer-grained control over whether a package > is opened, exported, or both is required then a normal module > declaration can be used. > > - This proposal is similar to one made by Stephen Colebourne [6], > in which he uses the term `exposes`, and to various unpublished > proposals (of which there have been many!). We here adopt the term > `opens` rather than `exposes` in order to avoid confusion between > the phonetically-similar `exports` and `exposes`, especially for > non-native speakers of English. We also use `open` to modify the > `module` keyword rather than introduce wildcards for the `opens` > directive, since the former is easier to explain. The latter would, > further, lead people to expect wildcards for the `exports` directive, > but we'd prefer to avoid that since it could encourage the careless > exportation of all of a module's packages. Exporting a package for > use as an API should always be an explicit, affirmative choice. > > - A separate `opens` directive, and the avoidance of wildcards, > simplifies the interactions amongst qualified and unqualified > package-level directives. We no longer need complex rules such as > those shown for the `private` modifier in the previous proposal [5]. > > - This proposal implies corresponding changes to the `ModuleDescriptor` > and `Module` APIs, but these are reasonably straightforward. > > - This proposal amends the proposal for #ResourceEncapsulation [6] > to replace the use of private exports with open modules and open > packages. Wherever that proposal indicates that a resource can be > located when its effective package name identifies a package that's > exported privately, assume instead that the resource can be located > when that package is either an element of an open module or an open > package in a normal module. > > - If a container is to ensure that a package in an application module > is available for deep reflection only by a trusted framework module > then it can arrange for that by rewriting that module's descriptor, > as suggested previously [8], to insert the appropriate qualified > `opens` directive. This works even in the case, pointed out by Jason > Greene [9], of two modules that open a package of the same name to > the same framework module, since opened and unexported packages are > not subject to the usual package-consistency checks. There is no > need any longer for the resolution algorithm to take this type of > scenario into account [a]. A more flexible way to arrange for such > precisely-constrained access is a separate issue and may be addressed > by a future proposal. > > - This proposal primarily addresses "friendly" uses of reflection, such > as dependency injection and persistence, in which the author of a > module knows in advance that one or more, or possibly all, packages > must be opened at run time for deep reflective access by frameworks. > Intrusive access to arbitrary packages of arbitrary modules by, e.g., > serialization frameworks or debugging tools, will still require the > use of sharp knives such as `--add-exports` or `--add-opens` > command-line options, the legacy unsupported `sun.misc.Unsafe` API > and related APIs, or JVM TI. > > - Using `--add-exports` or `--add-opens` command-line options, or their > equivalents, remains awkward, but sometimes they're the only way out. > To ease migration it's worth considering some way for an application > packaged as a JAR file to include such options in its `MANIFEST.MF` > file, as suggested by Simon Nash [b]. This is addressed in the > proposal for #AddExportsInManifest [c], which is hereby amended to > rename the `Add-Exports-Private` attribute to `Add-Opens`. > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes > [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation > [3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html > [4] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html > [5] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000390.html > [6] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000392.html > [7] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html > [8] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html > [9] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html > [a] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html > [b] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html > [c] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000391.html From david.lloyd at redhat.com Thu Nov 3 16:08:28 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 3 Nov 2016 11:08:28 -0500 Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports In-Reply-To: <20161027161537.655286378eggemoggin.niobe.net> References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net> <5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com> <20161011151514.A04ECD50FA@eggemoggin.niobe.net> <279bd57d-515a-a983-ffbb-6f569c29495e@redhat.com> <20161027161537.655286378eggemoggin.niobe.net> Message-ID: <3dd5cb13-f786-639d-703c-b225602f14aa@redhat.com> One question and one comment... On 10/27/2016 06:15 PM, mark.reinhold at oracle.com wrote: > The open-modules/packages proposal does not support your "Public Only / > Not Exported" mode, which corresponds to the missing "N S" row in the > compact table in [2], i.e., the "partly-encapsulated framework client" > case. Most reflective frameworks in use today will just reach in via > `setAccessible` whenever they need to, so whether the members of an > unexported package in a client module are public or private doesn't > really matter. The new proposal intentionally does not support this > case, which results in a simpler design. Isn't this essentially the same as doing a module.addReads(otherModule) though? IOW doesn't that essentially say that I can do public-only reflection on otherModule, even if it wasn't exported to me, or am I misunderstanding read edges (again)? >>> To confirm my understanding, would it be correct to say that your main >>> concern here is scenarios in which some of the classes in a user module >>> are intended to be manipulated reflectively by some trusted framework >>> module, but that framework module is either not known at compile time or >>> is otherwise inconvenient to name in the declaration of the user module? >> >> Correct. > > Good. I'll enter this as a new issue: > > #IndirectQualifiedReflectiveAccess --- Provide a means by which a > client module can grant qualified reflective access to a framework > module that is not known at compile time, assuming that the name of > some other module that represents the framework module is known. > A canonical example of this case is a client POJO module compiled > against a module that defines the JPA API. At run time the client > module, or a container or some other code acting on its behalf, > must grant qualified reflective access to its POJO packages to the > JPA implementation in actual use. Great. Also I just want to note that at this point we consider the proposal for #ClassLoaderNames to be acceptable, and that I'm still working on our reply to the new open modules proposal. -- - DML From mark.reinhold at oracle.com Thu Nov 3 23:32:52 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 03 Nov 2016 16:32:52 -0700 Subject: Proposal (revised^2): #ReflectiveAccessToNonExportedTypes & #AwkwardStrongEncapsulation: Open modules & open packages In-Reply-To: <1C97C371-66E1-4770-9307-FEE6CB912B81@paremus.com> References: <20161027155336.DBCE7DA329@eggemoggin.niobe.net> <1C97C371-66E1-4770-9307-FEE6CB912B81@paremus.com> Message-ID: <20161103163252.220277226> 2016/11/1 14:17:43 -0700, neil.bartlett at paremus.com: > ... > > With regards to this latest proposal for > #ReflectiveAccessToNonExportedTypes, it is certainly better than the > previous proposal (weak modules). However I find myself wondering > whether ?open? should be the default? I anticipate use-cases in which > a module author cannot predict whether or how the module will be > accessed reflectively ? in this case she would not have used the open > modifier and reflection will not be possible. As noted near the end, the proposal primarily addresses "friendly" uses of reflection, in which a module's author is aware (perhaps with help from an IDE) of which modules or packages need to be open for deep access by reflective frameworks. In scenarios where a module's author can't foresee the need for such access (e.g., intrusive serialization frameworks) then the framework's author must take more drastic measures (e.g., use the legacy unsupported unsafe API). This balance is intended to help maintain the integrity of a module as expressed by its author. > In fact I believe that > most module authors will open everything as a shortcut to > compatibility with existing frameworks. That may be true, at least in the near-to-medium term, but I wouldn't consider it a failure. > For example, DI frameworks such as Spring or CDI perform classpath > scanning to find annotated bean types. Web containers scan for > annotated Servlets or JAX-RS Resource classes. It may be hard for > developers using these framework to find the minimal set of packages > or modules to open. If a framework uses annotations then the presence of those annotations can help identify which packages (or entire modules) need to be open. We could formalize this by defining a meta-annotation which can be applied to a framework annotation in order to indicate that a package containing an element upon which that annotation is used is expected to be open, as previously suggested [1][2]. IDEs and other tools could interpret this annotation and offer to insert the necessary `opens` directives in the relevant module declaration, or even make the entire module `open`. Do you think such a meta-annotation would be worthwhile? > There may be a small number of utility packages > that can be closed, but wouldn't it be easier to explicitly close just > those? > > By analogy with other levels in the Java language: classes are not > final by default, and packages are not sealed by default. In my view these two points are, in retrospect, flaws in the original design of the language. That classes are not final by default has resulted in all sorts of gratuitous subclassing and, worse, some fairly nasty compatibility issues -- not least in Java itself! That packages are not sealed by default has resulted in split packages, whether intentional or not, and all the pain that they bring. Based upon our long-term experience with those decisions, I strongly suspect that if modules are open by default then most developers will define them that way at the start and never think about it again. If, on the other hand, they have to type `opens` to open a package or `open module` to open an entire module then it will be readily apparent that they're allowing additional access, which in turn will hopefully motivate them to think further about whether that's the right thing. Sometimes it is, and sometimes it isn't. > Finally, in the notes section of your proposal you recognise that it > will be common to have open modules. However in the main text there is > still some of the language about open modules being a migration step > towards ?normal? modules, which is reminiscent of your earlier > characterisation of weak modules. My concern with weak modules was > that they could never be fully eliminated, so it was wrong to talk > about them as purely transitional. Do you agree that there are valid > use-cases in which even a well-modularised application contains a > number of open modules? Yes, absolutely. Some modules will be open forever, and that's fine. Others will be open transitionally, on their way to becoming stronger normal modules. - Mark [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008644.html [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html From david.lloyd at redhat.com Mon Nov 7 15:08:15 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 7 Nov 2016 09:08:15 -0600 Subject: Proposal (revised^2): #ReflectiveAccessToNonExportedTypes & #AwkwardStrongEncapsulation: Open modules & open packages In-Reply-To: <20161027155336.DBCE7DA329@eggemoggin.niobe.net> References: <20161027155336.DBCE7DA329@eggemoggin.niobe.net> Message-ID: <836618d5-187b-e3d0-cae3-3b4d808c8d78@redhat.com> I was waiting on a response to an earlier question about readability, but I'm pretty sure I know the answer so I'm going to plow ahead. On 10/27/2016 10:53 AM, mark.reinhold at oracle.com wrote: > Extend the language of module declarations with the concept of _open_ > modules. Open modules make it easy to define modules whose internals > will be accessed at run time by reflection-based frameworks. [...] > Open modules can be considered an intermediate step on the migration > path, between automatic modules and normal modules: > > - An automatic module offers the traditional level of encapsulation: > All packages are both open for deep reflective access and exported > for ordinary compile-time and run-time access to their public types. > > - An open module offers a modest degree of stronger encapsulation: All > packages are open for deep reflective access, but the module's author > must specify which packages, if any, are exported for ordinary > compile-time and run-time access. > > - A normal module offers the strongest encapsulation: The module's > author must specify which packages, if any, are open, or exported, > or both. > > An open module is a good starting point for application code that's being > modularized by its author. It affords a separation of concerns, so that > the author can focus on defining the module's domain-specific API without > having to worry about how the module will be inspected or manipulated at > run time by reflective frameworks. This part looks OK. > To open specific packages in normal module declarations we introduce a > new per-package directive, `opens`. A package in a normal module can be > opened, or exported, or both, as follows: > > - If a package is only opened (`opens p;`) then properties (1), (2), > and (3) hold, i.e., its types are not accessible at compile time, > it does not affect resolution, and all of its elements are available > for deep reflection at run time. > > - If a package is only exported (`exports p;`) then those properties > do not hold, and it is exported exactly as it is today. > > - If a package is both exported and opened (`exports p; opens p;`) > then it is exported as today and is, additionally, available for > deep reflection at run time (3). I'd like to stop here and ask a bit about deep reflection, based on some discussion happening on jigsaw-dev. It was my assumption that "deep reflection" implies any indirect access to nonpublic members - which includes MethodHandle style access as well as classical reflection. So here are some questions: 1) I assume that one will continue to be able to unreflect any AccessibleObject which has had accessibility enabled and get MethodHandles which have deep reflective access. Is this assumption true? 2) Is it a goal to ultimately eliminate setAccessible()? In other words, is this the direction that Java security is moving? (No wrong answer here, I just haven't been able to glean a clear yes/no answer to this and it's definitely important to guide my thinking on this issue since I'm mentally trying to follow two paths at once on this.) 3) Given nonpublic member M in an opened package; will/could/should it be possible for Lookups (belonging to classes which observe the "openness" of M's package) to be able to directly create or acquire MethodHandles for M without using reflection (especially if the answer to #2 is "yes") and without relying on donated Lookups? Otherwise I think this part of the proposal as stated makes a lot of sense and covers our use cases fairly well. > It's possible to both open and export a package in a normal module, but > it's often inadvisable. This is especially so for API packages, since > normally an API's internal implementation details should be strongly > encapsulated. This combination of directives may, however, be useful for > legacy APIs whose internals are known to be accessed by existing code. > > To ensure the integrity of the platform we expect that all the modules of > the JDK itself will be normal modules and that very few, if any, packages > will be opened. > > The `opens` directive cannot be used in an `open` module, since all > packages in such modules are implicitly open. Good so far. > Both the `exports` and `opens` directives can be qualified, so that a > package is exported or opened only to certain other named modules. As > before, duplicate directives are not permitted in order to ensure easy > readability. At most one `exports` directive is relevant to any given > package, and at most one `opens` directive is relevant to any given > package. Do module names specified in qualified "opens" directives need to be present at compile or run time, or will absent module names be ignored? > There is no syntax for wildcards. If the defaults are not sufficient for > a package then the package must be named explicitly. OK > The existing syntax of `requires public` has long been confusing, so we > here take the opportunity to fix that problem by renaming the `public` > modifier in `requires` directives to `transitive`. Thus the declaration > > module foo.bar { > exports com.foo.bar; > requires public java.sql; > } > > is now written > > module foo.bar { > exports com.foo.bar; > requires transitive java.sql; > } Just to clarify, since this isn't really clearly spelled out in SOTMS or in any other doc at present (I think?), "requires transitive" means exactly that all packages exported by "java.sql" are also exported by "foo.bar", and that there is no way to narrow the set of packages in this case, correct? Also, I assume that requiring with "transitive" does not imply that "openness" is also transitively exposed, i.e. openness is always non-transitive. Is that a correct assumption? Overall this is a very good proposal, and I think (unless the answer to one of the above questions is very surprising) that we (Red Hat) are close to agreement on it. -- - DML From david.lloyd at redhat.com Tue Nov 8 14:30:01 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 8 Nov 2016 08:30:01 -0600 Subject: Quick issue update? Message-ID: <15c7d59c-de1c-57c6-9968-dbfb291d8868@redhat.com> Hi Mark, I was wondering if it would be possible to get a very quick rundown of which JPMS issues are currently being worked by Oracle engineers? And for anyone else as well: are there issues that you, or people that you represent, are currently working on? Thanks. -- - DML From tjwatson at us.ibm.com Tue Nov 8 14:37:44 2016 From: tjwatson at us.ibm.com (Thomas Watson) Date: Tue, 8 Nov 2016 08:37:44 -0600 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> References: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> Message-ID: > From: mark.reinhold at oracle.com > To: jpms-spec-experts at openjdk.java.net > Date: 10/31/2016 03:23 PM > Subject: Proposal: #NonHierarchicalLayers > Sent by: "jpms-spec-observers" > > Issue summary > ------------- > > #NonHierarchicalLayers --- Layers are presently constrained to be > hierarchical, i.e., each layer has at most one parent. Should this > restriction be relaxed so that a layer can have more than one parent? > Some have argued that this will be essential to the adoption of the > module system by a future version of the Java EE Platform. It would > also enable bidirectional interoperation with existing module systems > such as OSGi. [1] > > > Proposal > -------- > > Revise `java.lang.module.Configuration` and `java.lang.reflect.Layer` to > augment the existing instance methods that treat `this` as the parent of > the child being created with corresponding static methods that take a > list of parents. > > The list of the parents of a configuration can be thought of as a search > path. To construct a new configuration the resolver searches for modules > in the given parent configurations in the order in which they occur in > the list, and for any given parent it searches its parents recursively in > the same fashion, before moving on to the next parent in the list. This > search order is, thus, not only depth-first but also determined by the > order of the parents in each list. The latter property makes it easy to > tell when one candidate module will shadow another. > > To instantiate a new layer from a given configuration, the list of the > configurations of its parent layers must be identical to the list of the > parents of that configuration. > > The service providers for a module in a given layer are located in the > same way that the resolver searches for modules, i.e., depth-first and > in list order. > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#NonHierarchicalLayers > Overall the API works well for scenarios that map existing module systems to JPMS. I have questions about the new javadoc for the static methods on Layer and Configuration. I would have expected the javadoc for the static vs. the single parent non-static methods to be largely identical. But there seems to be more details on what causes errors on the static methods as well as additional API notes related to atomic behavior. Is this an oversight, or are there more error conditions for the static methods? Should the non-static methods be updated to state the behavior is equivalent to calling the static method with this Configuration/Layer as the single parent? Tom From mark.reinhold at oracle.com Tue Nov 8 14:55:57 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 08 Nov 2016 15:55:57 +0100 Subject: Quick issue update? In-Reply-To: <15c7d59c-de1c-57c6-9968-dbfb291d8868@redhat.com> References: <15c7d59c-de1c-57c6-9968-dbfb291d8868@redhat.com> Message-ID: <20161108155557.836763822> 2016/11/8 15:30:01 +0100, david.lloyd at redhat.com: > Hi Mark, I was wondering if it would be possible to get a very quick > rundown of which JPMS issues are currently being worked by Oracle > engineers? And for anyone else as well: are there issues that you, or > people that you represent, are currently working on? Sure. I'm at Devoxx BE this week but I'll post something when I can. - Mark From Alan.Bateman at oracle.com Wed Nov 9 12:58:43 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 9 Nov 2016 13:58:43 +0100 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: References: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> Message-ID: On 08/11/2016 15:37, Thomas Watson wrote: > : > Overall the API works well for scenarios that map existing module systems > to JPMS. I have questions about the new javadoc for the static methods on > Layer and Configuration. I would have expected the javadoc for the static > vs. the single parent non-static methods to be largely identical. But > there seems to be more details on what causes errors on the static methods > as well as additional API notes related to atomic behavior. Is this an > oversight, or are there more error conditions for the static methods? > > Should the non-static methods be updated to state the behavior is > equivalent to calling the static method with this Configuration/Layer as > the single parent? > The instance methods are specified to be equivalent to invoking the static methods with a single parent (to avoid repeating the javadoc) but I think we can make this a bit clearer. -Alan From mark.reinhold at oracle.com Mon Nov 14 20:42:14 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 14 Nov 2016 12:42:14 -0800 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: References: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> Message-ID: <20161114124214.294262324> 2016/11/8 6:37:44 -0800, Thomas Watson : > ... > > Overall the API works well for scenarios that map existing module systems > to JPMS. I have questions about the new javadoc for the static methods on > Layer and Configuration. I would have expected the javadoc for the static > vs. the single parent non-static methods to be largely identical. But > there seems to be more details on what causes errors on the static methods > as well as additional API notes related to atomic behavior. Is this an > oversight, or are there more error conditions for the static methods? No, there are not more error conditions for the static methods. > Should the non-static methods be updated to state the behavior is > equivalent to calling the static method with this Configuration/Layer as > the single parent? Yes. Alan has already made this update. - Mark From mark.reinhold at oracle.com Mon Nov 14 20:46:46 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 14 Nov 2016 12:46:46 -0800 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: References: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> Message-ID: <20161114124646.960356084> 2016/11/8 6:37:44 -0800, Thomas Watson : > ... > > Overall the API works well for scenarios that map existing module systems > to JPMS. I neglected to say in my first reply: Glad to hear this! Thanks for validating the proposal. - Mark From forax at univ-mlv.fr Wed Nov 16 09:48:19 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 16 Nov 2016 10:48:19 +0100 (CET) Subject: uses is useless ? Message-ID: <249470150.1626689.1479289699963.JavaMail.zimbra@u-pem.fr> Hi guys, sorry for the pun in the title. Last week end, i've taken a big codebase and try to retrofit it to use jigsaw, obviously the code has several (two in fact) cyclic dependencies and in both cases, i was able to break those cycles by changing the code to use services. In one case, i have wanted to write a small code that uses the ServiceLoader like DriverManager.drivers() to expose the service but if i do that, it's the module that contains that small code that has to declare the directive 'uses' inside its own module-info instead of being the code that effectively uses the service, which seems wrong to me *. So i think where missing an API point on ServiceLoader that allow to write a method that will ask for a directive uses on the behalf of the ServiceLoader. I think it's a good idea to leverage Lookup class here and add a method load(Class, Lookup) that load a service on the behalf to the lookup class and requires the module of the lookup class to declare a directive uses. cheers, R?mi * it's definitevly wrong because when i wanted to use jlink on the code to generate a kind of light version of the application, i was not able to know which modules was using the service to pass them as root modules to jlink. From forax at univ-mlv.fr Wed Nov 16 10:25:49 2016 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Wed, 16 Nov 2016 11:25:49 +0100 (CET) Subject: uses is useless ? In-Reply-To: <3435cfa6-92a6-ff87-ab63-d9c477bce3d9@oracle.com> References: <249470150.1626689.1479289699963.JavaMail.zimbra@u-pem.fr> <3435cfa6-92a6-ff87-ab63-d9c477bce3d9@oracle.com> Message-ID: <1651466605.1688207.1479291949878.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "Alan Bateman" > ?: "Remi Forax" > Cc: "jigsaw-dev" > Envoy?: Mercredi 16 Novembre 2016 11:09:40 > Objet: Re: uses is useless ? > On 16/11/2016 09:48, Remi Forax wrote: > >> Hi guys, >> sorry for the pun in the title. >> >> Last week end, i've taken a big codebase and try to retrofit it to use jigsaw, >> obviously the code has several (two in fact) cyclic dependencies and in both >> cases, i was able to break those cycles by changing the code to use services. > Good to hear this. > >> In one case, i have wanted to write a small code that uses the ServiceLoader >> like DriverManager.drivers() to expose the service but if i do that, >> it's the module that contains that small code that has to declare the directive >> 'uses' inside its own module-info instead of being the code that effectively >> uses the service, >> which seems wrong to me *. > I assume you are looking for Module::addUses which is useful when > libraries are using ServiceLoader on behalf of someone else. In these > scenarios then someone needs to declare that the `uses` of course, > otherwise the service provider modules won't be resolved. > > As regards the hygiene check in ServiceLoader then it is not strictly > needed but it has been very useful to identify cases where `uses` is > missing from the module declaration. The whole point of uses is to enable a kind of static analysis on the service dependencies, if 'uses' is not required, if you can easily bypass it or it doesn't provide useful information, then it should be removed from the spec. > >> >> So i think where missing an API point on ServiceLoader that allow to write a >> method that will ask for a directive uses on the behalf of the ServiceLoader. >> >> I think it's a good idea to leverage Lookup class here and add a method >> load(Class, Lookup) that load a service on the behalf to the lookup class and >> requires the module of the lookup class to declare a directive uses. >> >> cheers, >> R?mi >> >> * it's definitevly wrong because when i wanted to use jlink on the code to >> generate a kind of light version of the application, i was not able to know >> which modules was using the service to pass them as root modules to jlink. >> > jlink does not do service binding and you'll see in JEP 282 that there > is an open issue on whether to provide an option to help identify the > service provider modules to link into the image. Yes, jlink doesn't do that, but my tool on top of jlink does it :) But as i said above, if the information given by 'uses' do not allow to do static analysis like finding the root modules (services included), it perhaps means that 'uses' doesn't worth its own weight. > > -Alan R?mi From mark.reinhold at oracle.com Wed Nov 16 16:39:10 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 16 Nov 2016 08:39:10 -0800 (PST) Subject: Proposal (revised^2): #ReflectiveAccessToNonExportedTypes & #AwkwardStrongEncapsulation: Open modules & open packages In-Reply-To: <836618d5-187b-e3d0-cae3-3b4d808c8d78@redhat.com> References: <20161027155336.DBCE7DA329@eggemoggin.niobe.net> <836618d5-187b-e3d0-cae3-3b4d808c8d78@redhat.com> Message-ID: <20161116163910.D18841608F@eggemoggin.niobe.net> 2016/11/7 7:08:15 -0800, david.lloyd at redhat.com: > I was waiting on a response to an earlier question about readability, > but I'm pretty sure I know the answer so I'm going to plow ahead. > > On 10/27/2016 10:53 AM, mark.reinhold at oracle.com wrote: >> ... >> >> To open specific packages in normal module declarations we introduce a >> new per-package directive, `opens`. A package in a normal module can be >> opened, or exported, or both, as follows: >> >> - If a package is only opened (`opens p;`) then properties (1), (2), >> and (3) hold, i.e., its types are not accessible at compile time, >> it does not affect resolution, and all of its elements are available >> for deep reflection at run time. >> >> - If a package is only exported (`exports p;`) then those properties >> do not hold, and it is exported exactly as it is today. >> >> - If a package is both exported and opened (`exports p; opens p;`) >> then it is exported as today and is, additionally, available for >> deep reflection at run time (3). > > I'd like to stop here and ask a bit about deep reflection, based on some > discussion happening on jigsaw-dev. > > It was my assumption that "deep reflection" implies any indirect access > to nonpublic members - which includes MethodHandle style access as well > as classical reflection. So here are some questions: > > 1) I assume that one will continue to be able to unreflect any > AccessibleObject which has had accessibility enabled and get > MethodHandles which have deep reflective access. Is this assumption > true? Yes. This proposal does not change how method handles work. > 2) Is it a goal to ultimately eliminate setAccessible()? In other > words, is this the direction that Java security is moving? (No wrong > answer here, I just haven't been able to glean a clear yes/no answer to > this and it's definitely important to guide my thinking on this issue > since I'm mentally trying to follow two paths at once on this.) There is no specific plan as yet. In the long run we (as in, "many of us who work on the JDK") do want to move away from the difficult-to-secure identity-based core-reflection API and toward the easier-to-secure capability-based method-handle API, or perhaps to some new mirror-based API as yet undefined. > 3) Given nonpublic member M in an opened package; will/could/should it > be possible for Lookups (belonging to classes which observe the > "openness" of M's package) to be able to directly create or acquire > MethodHandles for M without using reflection (especially if the answer > to #2 is "yes") and without relying on donated Lookups? As things stand today, no. To support this would require defining a new `MethodHandles.Lookup` factory that grants deep access to packages that are open to the caller. We could do that or, as I mentioned earlier [1], we could take the simpler approach of requiring the client of a framework to pass (i.e., "donate") an appropriate Lookup object when initializing the framework. I gather from your comments on jigsaw-dev [2] that you'd prefer the former approach, and I agree that it'd require much less change on the part of framework users. I have an idea for how to do this which I'll write up soon, for #IndirectQualifiedReflectiveAccess. >> ... >> >> Both the `exports` and `opens` directives can be qualified, so that a >> package is exported or opened only to certain other named modules. As >> before, duplicate directives are not permitted in order to ensure easy >> readability. At most one `exports` directive is relevant to any given >> package, and at most one `opens` directive is relevant to any given >> package. > > Do module names specified in qualified "opens" directives need to be > present at compile or run time, or will absent module names be ignored? The latter, as noted in [3]: It is permitted for the `to` clause of an `exports` or `opens` statement to specify a module which is not observable. This is true in all phases. (If it weren't then we couldn't compile java.base, or run an image that contains just that module, since it has qualified exports to many other JDK modules.) >> ... >> >> The existing syntax of `requires public` has long been confusing, so we >> here take the opportunity to fix that problem by renaming the `public` >> modifier in `requires` directives to `transitive`. Thus the declaration >> >> module foo.bar { >> exports com.foo.bar; >> requires public java.sql; >> } >> >> is now written >> >> module foo.bar { >> exports com.foo.bar; >> requires transitive java.sql; >> } > > Just to clarify, since this isn't really clearly spelled out in SOTMS or > in any other doc at present (I think?), "requires transitive" means > exactly that all packages exported by "java.sql" are also exported by > "foo.bar" No, those packages are not in any sense exported by `foo.bar`. Remember that `requires`, and hence `requires transitive`, only affect readability. In the above example, any module that reads `foo.bar` will also be made to read `java.sql`, whether or not it does so explicitly itself. Readability, together with `exports` and `opens` directives, governs access control. A module that reads `foo.bar` will also read `java.sql`, and will thus have access to all of the exported packages of `java.sql`. > , and that there is no way to narrow the set of packages in > this case, correct? `requires transitive` isn't a re-export operator, so there's no way to narrow the set of packages conveyed in this manner. > Also, I assume that requiring with "transitive" does not imply that > "openness" is also transitively exposed, i.e. openness is always > non-transitive. Is that a correct assumption? No, though the details differ depending upon whether you start with core reflection or with method handles. Core reflection assumes readability (cf. #ReflectionWithoutReadability), so `requires` (and hence `requires transitive`) isn't really relevant. If a package is open to some module then code in that module can use core reflection and `setAccessible` to access private members regardless of readability relationships. The same would be true of a method handle unreflected from a core-reflection object upon which `setAccessible` has been invoked. A pure method-handles approach would, by contrast, respect readability. With the method-handl API as it stands today, if module A opens package P, and some other module B reads A, and code in B creates a `Lookup` object, then B will have reflective method-handle access to public (but not private) members of package P in A. B will read A if one of the following conditions is true: - B `requires` A - B `requires` C, and C `requires transitive` A, or - B does not `requires` A but somehow obtains a reads edge to A at run time (e.g., via `Module::addReads`). In the second case one can say that "openness" is conveyed transitively. > Overall this is a very good proposal, and I think (unless the answer to > one of the above questions is very surprising) that we (Red Hat) are > close to agreement on it. Glad to hear it. - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-November/009894.html [3] http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html From mark.reinhold at oracle.com Fri Nov 18 16:28:09 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 18 Nov 2016 08:28:09 -0800 (PST) Subject: Proposal: #ReadabilityAddedByLayerCreator Message-ID: <20161118162809.8241416ABE@eggemoggin.niobe.net> Issue summary ------------- #ReadabilityAddedByLayerCreator --- Provide a means by which the code that creates a layer can add readability edges from the modules in that layer to other modules, whether those modules are in that layer or in other layers. [1] Proposal -------- Revise `java.lang.reflect.Layer` so that the multi-parent layer-creation methods added for #NonHierarchicalLayers [2] return a capability object rather than the resulting layer. The capability will be represented by instances of a new nested class, `java.lang.reflect.Layer.Controller`, which will define an `addReads` method and also a method to retrieve the actual layer: public final static class Controller { /** Return the actual layer */ public Layer layer(); /** * Add the specified reads edge, if not already present. * Return this controller, so that invocations can be * chained. */ public Controller addReads(Module source, Module target); } Typical usage would be: Layer.Controller lc = Layer.defineModules(cf, parents, clf); Module m1 = lc.layer().findModule("m1").get(); Module m2 = otherLayer.findModule("m2").get(); lc.addReads(m1, m2); // Ensures m1.canRead(m2) == true The source module must be in the controller's layer; the target module can be in any layer. This method does nothing if the source module can already read the target module. Like all capability objects, instances of `Layer.Controller` must be handled with care since leaking them could enable malicious use. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReadabilityAddedByLayerCreator [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000442.html From mark.reinhold at oracle.com Fri Nov 18 16:29:09 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 18 Nov 2016 08:29:09 -0800 (PST) Subject: Proposal: #IndirectQualifiedReflectiveAccess Message-ID: <20161118162909.879D016AC0@eggemoggin.niobe.net> Issue summary ------------- #IndirectQualifiedReflectiveAccess --- Provide a means by which a client module can grant qualified reflective access to a framework module that is not known at compile time, assuming that the name of some other module that represents the framework module is known. A canonical example of this case is a client POJO module compiled against a module that defines the JPA API. At run time the client module, or a container or some other code acting on its behalf, must grant qualified reflective access to its POJO packages to the JPA implementation in actual use. [1] Proposal -------- This proposal has three parts. As a running example we assume a JPA client POJO module declared thus: module foo.model { requires java.persistence; opens com.foo.model to java.persistence; } where `java.persistence` is the name of the JPA API module and `com.foo.model` is the package that contains the POJO classes. JPA is an example of an _abstract reflective framework_. Its API is defined in one module but its implementations, of which there can be more than one, are defined in other modules known only at run time. To address this issue we must provide a way for such frameworks, or the containers in which they run, to convey qualified access to the open packages of client modules to the framework implementation modules. * * * First, to support abstract reflective frameworks when used in ordinary Java SE applications, outside of a container environment, we revise the specification of `java.lang.reflect.Module::addOpens` so that if module A has a qualified `opens` of package P to module B, then code in B can invoke this method to open P to any other module. When running outside a container a JPA entity manager is created via one of the `javax.persistence.Persistence::createEntityManagerFactory` methods, which locates and initializes a suitable persistence provider. As part of that process it can use the `addOpens` method on the client module to open the `com.foo.model` package to the provider's module. This will work since the `foo.model` module opens that package to the `java.persistence`' module. * * * Second, to support abstract reflective frameworks when used inside a container, where the container itself rather than the framework module locates and initializes framework implementations, we extend the `Layer.Controller` class proposed for #ReadabilityAddedByLayerCreator [2] with an `addOpens` method: public final static class Controller { ... /** * Open a package in the source module, which must be in this * layer, to the target module, which can be in any layer. */ public Controller addOpens(Module source, String pkg, Module target); } This allows the container to load the client POJO module into a layer, find the packages of that module that are already open to the framework module, and then open those same packages to the framework implementation modules. (The container has the ability to open arbitrary packages to arbitrary modules, of course, but it should use this power with great care.) * * * Third, and finally, to allow framework authors to adopt method handles in preference to, or in place of, the `java.lang.reflect` core reflection API and its `setAccessible` method, we define a single new method in `java.lang.invoke.MethodHandles`: public class MethodHandles { ... Lookup privateLookupIn(Class target, Lookup lookup); } If the given target class, say C, is in package P, and P is either open to the module in which the given lookup object's class is defined or P is in that module, then the result is a new lookup object for C with private access to all members of C. Typical usage in framework code would be: Object entity = ...; long id = ...; Lookup l = MethodHandles.privateLookupIn(entity.getClass(), MethodHandles.lookup()); l.findSetter(entity.getClass(), "id", Long.TYPE).invokeExact(entity, id); A new lookup object must be created for each client class manipulated in this way, but these are inexpensive so this should not be a problem. Notes ----- - This proposal requires maintainers of abstract reflective frameworks, and of container applications, to make modest changes in order to work well in a modular setting. If an existing framework module must be used without change then its client modules must open all of the relevant packages without qualification, or else other arrangements must be made to provide the requisite access. The same is true in the case of an abstract reflective framework that's purely an API, with no code to locate and initialize a suitable implementation, when running outside of a container. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#IndirectQualifiedReflectiveAccess [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-November/000456.html From mark.reinhold at oracle.com Fri Nov 18 16:30:09 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 18 Nov 2016 08:30:09 -0800 (PST) Subject: Discussion: #MutableConfigurations, #LazyConfigurationAndInstantiation, #CyclicDependences, & #DiscardableModules Message-ID: <20161118163009.8C54A16AC5@eggemoggin.niobe.net> References: http://openjdk.java.net/projects/jigsaw/spec/issues/#MutableConfigurations http://openjdk.java.net/projects/jigsaw/spec/issues/#LazyConfigurationAndInstantiation http://openjdk.java.net/projects/jigsaw/spec/issues/#CyclicDependences http://openjdk.java.net/projects/jigsaw/spec/issues/#DiscardableModules In light of Thomas Watson's initial attempt to achieve bidirectional interoperation between OSGi and JPMS [1], my subsequent suggestion that this can be done indirectly with the present design by modeling a dynamic module as a sequence of JPMS modules over time [2] if given a solution to #NonHierarchicalLayers [3], and then Watson's validation of that approach [4], it appears that we no longer have any need to address the above four issues, at least as far as OSGi is concerned. David: Is this approach workable for JBoss Modules as well? If so then I'd like to close these issues out; if not then I'd like to understand if there are additional, smaller changes that would make this approach acceptable while avoiding the complexity of complete solutions to these issues. - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-August/000062.html, or http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000410.html [3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000442.html [4] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-November/000449.html From tjwatson at us.ibm.com Fri Nov 18 18:32:31 2016 From: tjwatson at us.ibm.com (Thomas Watson) Date: Fri, 18 Nov 2016 12:32:31 -0600 Subject: Proposal: #ReadabilityAddedByLayerCreator In-Reply-To: <20161118162809.8241416ABE@eggemoggin.niobe.net> References: <20161118162809.8241416ABE@eggemoggin.niobe.net> Message-ID: I think this proposal will work fine for the usage I have in mind. I like the approach of a Controller because it provides a place to put other commands which a framework could use to control the modules within the Layer. I see you have taken advantage of that in other proposals. I look forward to trying out the API when it gets into a build. Thanks Tom > From: mark.reinhold at oracle.com > To: jpms-spec-experts at openjdk.java.net > Date: 11/18/2016 10:28 AM > Subject: Proposal: #ReadabilityAddedByLayerCreator > Sent by: "jpms-spec-observers" > > Issue summary > ------------- > > #ReadabilityAddedByLayerCreator --- Provide a means by which the code > that creates a layer can add readability edges from the modules in that > layer to other modules, whether those modules are in that layer or in > other layers. [1] > > Proposal > -------- > > Revise `java.lang.reflect.Layer` so that the multi-parent layer-creation > methods added for #NonHierarchicalLayers [2] return a capability object > rather than the resulting layer. The capability will be represented by > instances of a new nested class, `java.lang.reflect.Layer.Controller`, > which will define an `addReads` method and also a method to retrieve the > actual layer: > > public final static class Controller { > > /** Return the actual layer */ > public Layer layer(); > > /** > * Add the specified reads edge, if not already present. > * Return this controller, so that invocations can be > * chained. > */ > public Controller addReads(Module source, Module target); > > } > > Typical usage would be: > > Layer.Controller lc = Layer.defineModules(cf, parents, clf); > Module m1 = lc.layer().findModule("m1").get(); > Module m2 = otherLayer.findModule("m2").get(); > lc.addReads(m1, m2); // Ensures m1.canRead(m2) == true > > The source module must be in the controller's layer; the target module > can be in any layer. This method does nothing if the source module can > already read the target module. > > Like all capability objects, instances of `Layer.Controller` must be > handled with care since leaking them could enable malicious use. > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ > #ReadabilityAddedByLayerCreator > [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016- > October/000442.html > From mark.reinhold at oracle.com Sat Nov 19 00:28:21 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 18 Nov 2016 16:28:21 -0800 Subject: Closing out more open issues Message-ID: <20161118162821.56770200@eggemoggin.niobe.net> Proposals for the following issues have been available for evaluation and experimentation for some time [1]: #AddExportsInManifest #ClassFilesAsResources #ClassLoaderNames #ResourceEncapsulation #ServiceLoaderEnhancements #VersionsInModuleNames #ClassLoaderNames Most responses to these proposals have been either neutral or positive. There are some concerns about the impact of #VersionsInModuleNames on the naming of automatic modules, but I'd like to move forward with it as-is and see what more we learn once it's available in the mainline JDK 9 EA builds. If no EG member objects by this time next Friday, 25 November, then I'll close these issues out. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ From david.lloyd at redhat.com Mon Nov 21 15:01:27 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 21 Nov 2016 09:01:27 -0600 Subject: Discussion: #MutableConfigurations, #LazyConfigurationAndInstantiation, #CyclicDependences, & #DiscardableModules In-Reply-To: <20161118163009.8C54A16AC5@eggemoggin.niobe.net> References: <20161118163009.8C54A16AC5@eggemoggin.niobe.net> Message-ID: <8e5bd410-9737-db44-6c2f-4289ad4c8db2@redhat.com> On 11/18/2016 10:30 AM, mark.reinhold at oracle.com wrote: > References: > > http://openjdk.java.net/projects/jigsaw/spec/issues/#MutableConfigurations > http://openjdk.java.net/projects/jigsaw/spec/issues/#LazyConfigurationAndInstantiation > http://openjdk.java.net/projects/jigsaw/spec/issues/#CyclicDependences > http://openjdk.java.net/projects/jigsaw/spec/issues/#DiscardableModules > > In light of Thomas Watson's initial attempt to achieve bidirectional > interoperation between OSGi and JPMS [1], my subsequent suggestion that > this can be done indirectly with the present design by modeling a dynamic > module as a sequence of JPMS modules over time [2] if given a solution to > #NonHierarchicalLayers [3], and then Watson's validation of that approach > [4], it appears that we no longer have any need to address the above four > issues, at least as far as OSGi is concerned. > > David: Is this approach workable for JBoss Modules as well? If so then > I'd like to close these issues out; if not then I'd like to understand > if there are additional, smaller changes that would make this approach > acceptable while avoiding the complexity of complete solutions to these > issues. Partially, but not completely... In particular I am worried about #CyclicDependences (and honestly I'm not entirely convinced that OSGi works completely correctly without it either, for the same reasons that we would need it; I think maybe Mr. Watson's implementation can skate by on an illusion of circularity but I can't see how it would work for us). I am concerned that it will not be realistic to prune every circular reference in any potential module graph, even using service loaders; IIRC I had to add circularity support to JBoss Modules almost immediately due to cases that could not be practically supported in any other way, and I don't expect that to have changed now. This is especially true where compile time dependencies differ from run time dependencies, which I think will be fairly common, due to the basic reality (upon which I have expounded in the past) that the minimum practical de facto dependency contract of a given artifact is defined by its ABI, not by its identity or other content. Some module circuits are fairly long and it's not obvious how they could be resolved. Also I am having a hard time figuring out how to adapt our dependency API to the #NonHierarchicalLayers proposal. I'll post a separate reply to that mail though. Beyond that, I would need to do more testing to see what blows up, but the other portions seem acceptable and practical. -- - DML From david.lloyd at redhat.com Mon Nov 21 15:17:14 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 21 Nov 2016 09:17:14 -0600 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> References: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> Message-ID: <0f23ba26-9e93-a62b-b537-be0d5a91b895@redhat.com> On 10/31/2016 03:22 PM, mark.reinhold at oracle.com wrote: > Issue summary > ------------- > > #NonHierarchicalLayers --- Layers are presently constrained to be > hierarchical, i.e., each layer has at most one parent. Should this > restriction be relaxed so that a layer can have more than one parent? > Some have argued that this will be essential to the adoption of the > module system by a future version of the Java EE Platform. It would > also enable bidirectional interoperation with existing module systems > such as OSGi. [1] > > > Proposal > -------- > > Revise `java.lang.module.Configuration` and `java.lang.reflect.Layer` to > augment the existing instance methods that treat `this` as the parent of > the child being created with corresponding static methods that take a > list of parents. > > The list of the parents of a configuration can be thought of as a search > path. To construct a new configuration the resolver searches for modules > in the given parent configurations in the order in which they occur in > the list, and for any given parent it searches its parents recursively in > the same fashion, before moving on to the next parent in the list. This > search order is, thus, not only depth-first but also determined by the > order of the parents in each list. The latter property makes it easy to > tell when one candidate module will shadow another. > > To instantiate a new layer from a given configuration, the list of the > configurations of its parent layers must be identical to the list of the > parents of that configuration. > > The service providers for a module in a given layer are located in the > same way that the resolver searches for modules, i.e., depth-first and > in list order. I am having difficulty adapting our dependency API to it. We use the ability to add a dependency link from a module in one namespace to a module in another. This ability was explicitly added after realizing that using a search path made the usage parallel namespaces without name mangling impossible. In JBoss Modules this is accomplished by using a module loader (reference)/name (string) tuple for each dependency, but AFAICT, because of Jigsaw's reliance on a bytecode image instead of a programmatic API to define modules, a similar API wherein we give a Layer plus module name to establish the links seems impossible. Maybe we could have a way to reference each "parent" layer by a "relative name", which would only be "visible" to the referencing layer, that can be used to select which layer a dependency should come from (I think we'd need to either support circularity in layers, or allow dependencies to unrelated layers, for this to fully work for us though)? Any other suggestions on this are welcome as I'm at a dead end. -- - DML From mark.reinhold at oracle.com Mon Nov 21 21:52:01 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 21 Nov 2016 13:52:01 -0800 Subject: Discussion: #MutableConfigurations, #LazyConfigurationAndInstantiation, #CyclicDependences, & #DiscardableModules In-Reply-To: <8e5bd410-9737-db44-6c2f-4289ad4c8db2@redhat.com> References: <20161118163009.8C54A16AC5@eggemoggin.niobe.net> <8e5bd410-9737-db44-6c2f-4289ad4c8db2@redhat.com> Message-ID: <20161121135201.534880609@eggemoggin.niobe.net> 2016/11/21 7:01:27 -0800, david.lloyd at redhat.com: > On 11/18/2016 10:30 AM, mark.reinhold at oracle.com wrote: >> ... >> >> David: Is this approach workable for JBoss Modules as well? If so then >> I'd like to close these issues out; if not then I'd like to understand >> if there are additional, smaller changes that would make this approach >> acceptable while avoiding the complexity of complete solutions to these >> issues. > > Partially, but not completely... In particular I am worried about > #CyclicDependences (and honestly I'm not entirely convinced that OSGi > works completely correctly without it either, for the same reasons that > we would need it; I think maybe Mr. Watson's implementation can skate by > on an illusion of circularity but I can't see how it would work for us). > I am concerned that it will not be realistic to prune every circular > reference in any potential module graph, even using service loaders; > IIRC I had to add circularity support to JBoss Modules almost > immediately due to cases that could not be practically supported in any > other way, and I don't expect that to have changed now. This is > especially true where compile time dependencies differ from run time > dependencies, which I think will be fairly common, due to the basic > reality (upon which I have expounded in the past) that the minimum > practical de facto dependency contract of a given artifact is defined by > its ABI, not by its identity or other content. Some module circuits are > fairly long and it's not obvious how they could be resolved. If you're using an approach similar to what I suggested for OSGi [1] then you should be able to implement cycles amongst JBoss modules by inserting the necessary readability edges after each module is instantiated in its own layer. This will be much easier with the new API proposed for #ReadabilityAddedByLayerCreator [2]. > Also I am having a hard time figuring out how to adapt our dependency > API to the #NonHierarchicalLayers proposal. I'll post a separate reply > to that mail though. (I'll reply to that in a moment.) > Beyond that, I would need to do more testing to see what blows up, but > the other portions seem acceptable and practical. Okay, good. - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000410.html [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-November/000456.html From mark.reinhold at oracle.com Mon Nov 21 21:56:38 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 21 Nov 2016 13:56:38 -0800 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: <0f23ba26-9e93-a62b-b537-be0d5a91b895@redhat.com> References: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> <0f23ba26-9e93-a62b-b537-be0d5a91b895@redhat.com> Message-ID: <20161121135638.309938232@eggemoggin.niobe.net> 2016/11/21 7:17:14 -0800, david.lloyd at redhat.com: > On 10/31/2016 03:22 PM, mark.reinhold at oracle.com wrote: >> ... >> >> Proposal >> -------- >> >> Revise `java.lang.module.Configuration` and `java.lang.reflect.Layer` to >> augment the existing instance methods that treat `this` as the parent of >> the child being created with corresponding static methods that take a >> list of parents. >> >> ... > > I am having difficulty adapting our dependency API to it. We use the > ability to add a dependency link from a module in one namespace to a > module in another. This ability was explicitly added after realizing > that using a search path made the usage parallel namespaces without name > mangling impossible. In JBoss Modules this is accomplished by using a > module loader (reference)/name (string) tuple for each dependency, but > AFAICT, because of Jigsaw's reliance on a bytecode image instead of a > programmatic API to define modules, a similar API wherein we give a > Layer plus module name to establish the links seems impossible. You can construct `java.lang.module.ModuleDescriptor` objects via the API, and then use those to configure layers; you don't always need a `module-info.class` file. Layers don't have names, but if you're using the approach I suggested for OSGi [1] then it shouldn't matter, since every JBoss module will be in its own layer. (If you really do need to give layers separate names then you could do so in a `WeakHashMap` that you maintain on the side, and have your resolver refer to that.) - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000410.html From mark.reinhold at oracle.com Mon Nov 21 22:43:31 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 21 Nov 2016 14:43:31 -0800 Subject: Issue summary now lists issues by both category and status Message-ID: <20161121144331.161298047@eggemoggin.niobe.net> I've enhanced the issue summary to list issues by category, as before, and also by status, so that it's easy to see where we are: http://openjdk.java.net/projects/jigsaw/spec/issues/ I expect to post proposals or start discussion threads for all of the remaining open issues during the next few weeks. - Mark From mark.reinhold at oracle.com Tue Nov 22 16:47:35 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 22 Nov 2016 08:47:35 -0800 (PST) Subject: Proposal: #ClassFileModuleName Message-ID: <20161122164735.6899C1893F@eggemoggin.niobe.net> Issue summary ------------- #ClassFileModuleName --- The name of a module is not a simple UTF-8 string but is, rather, derived from the value of the `this_class` field of the `ClassFile` structure, which is awkward. [1] Proposal -------- Revise the binary form of module declarations [2] as follows: - Add a `u2 module_name_index` field to the `Module` class-file attribute, immediately preceding the existing `module_flags` field. The value of this field will be the index of a `CONSTANT_Utf8_info` structure in the constant pool that represents the module's name in internal form, i.e., with period characters replaced by slashes [3]. - Rather than encode the module's name in the value of the `this_class` field of the `ClassFile` structure, simply set this field to zero. (Side note: The prototype implementation does not store module names in internal form, even though the draft specification [2] has long mandated that. The implementation will be fixed.) [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassFileModuleName [2] http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html [3] http://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.2.1 From mark.reinhold at oracle.com Tue Nov 22 16:48:35 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 22 Nov 2016 08:48:35 -0800 (PST) Subject: Proposal: #ClassFileAccModule Message-ID: <20161122164835.6F47218943@eggemoggin.niobe.net> Issue summary ------------- #ClassFileAccModule --- The `ACC_MODULE` constant is currently specified to have the value 0x8000. This is the last available bit remaining across all of the various `access_flags` fields of a class file, and thus should be reserved for some unspecified future purpose where it may be useful to use the same value in all such fields. Alternative candidates for `ACC_MODULE` include 0x0040 (overlaps with `ACC_VOLATILE` and `ACC_BRIDGE`) and 0x0080 (`ACC_TRANSIENT` and `ACC_VARARGS`). [1] Proposal -------- Do not change the value of this constant. The high-order bits of the `access_flags` field have long been used to characterize the unusual, non-class nature of some class files: - `ACC_ANNOTATION` (0x2000) indicates an annotation type, even though `ACC_ANNOTATION` does not apply to fields or methods, and - `ACC_ENUM` (0x4000) indicates an `enum` type, even though `ACC_ENUM` does not apply to methods. A binary module descriptor is a new kind of unusual class file, hence we have done the obvious thing and allocated the high-order bit 0x8000 for `ACC_MODULE`. We could try to be more clever here, but that would require attempting to predict the future. There are still plenty of other ways to encode potential future features; a value class could, e.g., be indicated by 0x0100. In the worst case the `access_flags` field could simply be made wider in some future version of the class-file specification. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassFileAccModule From mark.reinhold at oracle.com Tue Nov 22 16:49:35 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 22 Nov 2016 08:49:35 -0800 (PST) Subject: Proposal: #ModuleNameCharacters Message-ID: <20161122164935.7633C18948@eggemoggin.niobe.net> Issue summary ------------- #ModuleNameCharacters --- Module names are presently constrained to be Java identifiers. Some existing module systems allow additional characters in module names, such as hyphens and slashes. Should this restriction be lifted or, perhaps, should it somehow be made layer-specific? [1] Proposal -------- Make no changes here. Modules are a new construct of the Java programming language in the present design. In the source language they are hence identified by qualified names [2] in the same manner as the existing structural constructs, i.e., packages and classes. As such these names do allow some unusual characters, though not hyphens or slashes [3]. Module names in compiled module-declaration class files are recorded in `CONSTANT_Utf8_info` structures, and thus have fewer constraints. They replace periods (`'.'`) with forward slashes (`'/'`), and disallow periods, semicolons (`';'`), and left square brackets (`'['`) [4]. The `ModuleDescriptor` API can read class files that contain module names not expressible in the source language, though it cannot be used to construct instances with such names. Artifacts that define modules with such names will be processed normally when placed on the module path. The present design is, then, consistent with the existing treatment of qualified names in the language, in class files, and in the Java SE API. A different module system with a more-flexible naming scheme can easily refer to JPMS modules, per the agreed interoperation requirement [5]. The requirements do not mandate bidirectional interoperation, which for this issue would mean that JPMS modules must be able to refer to non-JPMS modules with non-JPMS names. To support that would add significant complexity to this specification and its implementations. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameCharacters [2] http://docs.oracle.com/javase/specs/jls/se8/html/jls-6.html#jls-6.2 [3] http://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html#jls-3.8 [4] http://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.2.1 [5] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation From david.lloyd at redhat.com Tue Nov 22 21:05:13 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 22 Nov 2016 15:05:13 -0600 Subject: Proposal: #ModuleNameCharacters In-Reply-To: <20161122164935.7633C18948@eggemoggin.niobe.net> References: <20161122164935.7633C18948@eggemoggin.niobe.net> Message-ID: <3c1891c8-b8c5-c1ba-ec87-34b7824fd7f7@redhat.com> On 11/22/2016 10:49 AM, mark.reinhold at oracle.com wrote: > Proposal > -------- > > Make no changes here. TL;DR: we can't accept this proposal as-is. Expounding more below. > Modules are a new construct of the Java programming language in the > present design. In the source language they are hence identified by > qualified names [2] in the same manner as the existing structural > constructs, i.e., packages and classes. As such these names do allow > some unusual characters, though not hyphens or slashes [3]. > > Module names in compiled module-declaration class files are recorded in > `CONSTANT_Utf8_info` structures, and thus have fewer constraints. I believe that according to the JVM spec (until now anyway), this field type by itself has no constraints at all, beyond being a valid "modified" UTF-8 string. > They replace periods (`'.'`) with forward slashes (`'/'`), and disallow > periods, semicolons (`';'`), and left square brackets (`'['`) [4]. These name manglings are really just plain weird in this context, and are clearly an implementation artifact. How did we arrive at this place? I feel like it springs from the long-maligned conflation of module descriptors with class files. But modules are not types and AFAICT these restrictions really make no sense from any other perspective than one of implementation. > The `ModuleDescriptor` API can read class files that contain module names not > expressible in the source language, though it cannot be used to construct > instances with such names. This restriction seems arbitrary, but we can mitigate it I guess. > Artifacts that define modules with such names > will be processed normally when placed on the module path. OK. Just to be clear though, the use cases that we are dealing with would not involve a module path. > The present design is, then, consistent with the existing treatment of > qualified names in the language, in class files, and in the Java SE API. I do not believe that any of these statements is sufficient to justify the constraint... more below. > A different module system with a more-flexible naming scheme can easily > refer to JPMS modules, per the agreed interoperation requirement [5]. > The requirements do not mandate bidirectional interoperation, which for > this issue would mean that JPMS modules must be able to refer to non-JPMS > modules with non-JPMS names. This is also not sufficient to justify the constraint, though it does help to explain the reasoning why the constraint existed in the first place. > To support that would add significant > complexity to this specification and its implementations. I am sympathetic to this, but I think it needs more discussion, particularly as the proposed complexities have not been explained. The overarching problem here for us is that the JPMS specifies a new deployment model which is being put forth as a standard for future development. However we can't accept a solution which will render existing, otherwise satisfactory, alternative modular deployment models unusable with this new model: for example OSGi, Java EE, or JBoss Modules and its usages in various contexts. To do so would be, at a minimum, a huge disservice to our users, and would require a large investment to migrate to the new model: a lossy move, because in this case we'd be losing features by definition. Thus one implicit requirement that we have is that we can successfully evolve our deployment models into the JPMS and interoperate fully; else we must choose between sentencing our own deployment models to death or dismemberment, or else being unable to take advantage of the JPMS capabilities (and in either case we would have no motivation to support it, though for different reasons in each case). While we can theoretically support some name constraints on our model (leaving Java EE aside), the constraints we could support do not match the ones that exist, and therefore this information doesn't really help much. But the other implicit requirement that we have is to ensure that it's possible to adapt Java EE in some reasonable manner. It's my belief that in order to do so we need to be able to create modules with names that match the current constraints for Java EE module names (i.e. effectively no constraint, just valid UTF-8). Name mangling to accommodate this (which would be our only other option) is unnecessarily user-unfriendly at best, and outright incompatible at worst. I think we should talk about the factors that are complicating this issue and proceed from there... maybe there's something we can do to help. -- - DML From david.lloyd at redhat.com Tue Nov 22 21:19:19 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 22 Nov 2016 15:19:19 -0600 Subject: Discussion: #MutableConfigurations, #LazyConfigurationAndInstantiation, #CyclicDependences, & #DiscardableModules In-Reply-To: <20161121135201.534880609@eggemoggin.niobe.net> References: <20161118163009.8C54A16AC5@eggemoggin.niobe.net> <8e5bd410-9737-db44-6c2f-4289ad4c8db2@redhat.com> <20161121135201.534880609@eggemoggin.niobe.net> Message-ID: On 11/21/2016 03:52 PM, mark.reinhold at oracle.com wrote: > 2016/11/21 7:01:27 -0800, david.lloyd at redhat.com: >> On 11/18/2016 10:30 AM, mark.reinhold at oracle.com wrote: >>> ... >>> >>> David: Is this approach workable for JBoss Modules as well? If so then >>> I'd like to close these issues out; if not then I'd like to understand >>> if there are additional, smaller changes that would make this approach >>> acceptable while avoiding the complexity of complete solutions to these >>> issues. >> >> Partially, but not completely... In particular I am worried about >> #CyclicDependences (and honestly I'm not entirely convinced that OSGi >> works completely correctly without it either, for the same reasons that >> we would need it; I think maybe Mr. Watson's implementation can skate by >> on an illusion of circularity but I can't see how it would work for us). >> I am concerned that it will not be realistic to prune every circular >> reference in any potential module graph, even using service loaders; >> IIRC I had to add circularity support to JBoss Modules almost >> immediately due to cases that could not be practically supported in any >> other way, and I don't expect that to have changed now. This is >> especially true where compile time dependencies differ from run time >> dependencies, which I think will be fairly common, due to the basic >> reality (upon which I have expounded in the past) that the minimum >> practical de facto dependency contract of a given artifact is defined by >> its ABI, not by its identity or other content. Some module circuits are >> fairly long and it's not obvious how they could be resolved. > > If you're using an approach similar to what I suggested for OSGi [1] > then you should be able to implement cycles amongst JBoss modules by > inserting the necessary readability edges after each module is > instantiated in its own layer. This will be much easier with the new > API proposed for #ReadabilityAddedByLayerCreator [2]. I asked earlier whether this allows the class resolver to resolve against modules with a lately-added readability edge as you describe here, and it sounded like the answer was "no"; did I miss something? If my understanding was right, then this doesn't allow for circularity at the class level where we need it. >> Also I am having a hard time figuring out how to adapt our dependency >> API to the #NonHierarchicalLayers proposal. I'll post a separate reply >> to that mail though. > > (I'll reply to that in a moment.) > >> Beyond that, I would need to do more testing to see what blows up, but >> the other portions seem acceptable and practical. > > Okay, good. > > - Mark > > > [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000410.html > [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-November/000456.html > -- - DML From david.lloyd at redhat.com Tue Nov 22 21:23:34 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 22 Nov 2016 15:23:34 -0600 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: <20161121135638.309938232@eggemoggin.niobe.net> References: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> <0f23ba26-9e93-a62b-b537-be0d5a91b895@redhat.com> <20161121135638.309938232@eggemoggin.niobe.net> Message-ID: <0a1e41c6-9d36-ad5a-5ad0-7445e7ae3883@redhat.com> On 11/21/2016 03:56 PM, mark.reinhold at oracle.com wrote: > 2016/11/21 7:17:14 -0800, david.lloyd at redhat.com: >> On 10/31/2016 03:22 PM, mark.reinhold at oracle.com wrote: >>> ... >>> >>> Proposal >>> -------- >>> >>> Revise `java.lang.module.Configuration` and `java.lang.reflect.Layer` to >>> augment the existing instance methods that treat `this` as the parent of >>> the child being created with corresponding static methods that take a >>> list of parents. >>> >>> ... >> >> I am having difficulty adapting our dependency API to it. We use the >> ability to add a dependency link from a module in one namespace to a >> module in another. This ability was explicitly added after realizing >> that using a search path made the usage parallel namespaces without name >> mangling impossible. In JBoss Modules this is accomplished by using a >> module loader (reference)/name (string) tuple for each dependency, but >> AFAICT, because of Jigsaw's reliance on a bytecode image instead of a >> programmatic API to define modules, a similar API wherein we give a >> Layer plus module name to establish the links seems impossible. > > You can construct `java.lang.module.ModuleDescriptor` objects via the > API, and then use those to configure layers; you don't always need a > `module-info.class` file. > > Layers don't have names, but if you're using the approach I suggested > for OSGi [1] then it shouldn't matter, since every JBoss module will be > in its own layer. (If you really do need to give layers separate names > then you could do so in a `WeakHashMap` that you maintain on the side, > and have your resolver refer to that.) No, I was just suggesting a solution to the problem I see. To degenerate the problem to the simplest case: I may have to layers, A and B, which are unrelated and both contain a module X. I may need the module X in one layer to have a dependency on the module X in another layer. Unless there's new code I haven't seen, each dependency is expressed by name only, therefore this is impossible. Whereas in our system, a dependency is a tuple of (module loader, module name), allowing arbitrary weaving of modules from different name spaces with no problem. I don't think we need it to work the exact same way, but we need some equivalent. -- - DML From forax at univ-mlv.fr Wed Nov 23 19:59:58 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 23 Nov 2016 20:59:58 +0100 (CET) Subject: Proposal: #ModuleNameCharacters In-Reply-To: <3c1891c8-b8c5-c1ba-ec87-34b7824fd7f7@redhat.com> References: <20161122164935.7633C18948@eggemoggin.niobe.net> <3c1891c8-b8c5-c1ba-ec87-34b7824fd7f7@redhat.com> Message-ID: <2034282380.1641047.1479931198355.JavaMail.zimbra@u-pem.fr> I agree with David, module names are just names so there should be encoded as 'UTF8' kind of constant with no restriction. Java (the language) module name format has more restrictions because javac has to compile module-info.java files, but in the bytecode there is no need to encode module name as 'internal name'. When doing the JSR 292 spec, we (the EG) removed all the constraints on class name, method name, etc that could be removed, in order to ease the mapping of any languages on top of the VM. If we want ease the mapping between any existing and future module system to JPMS, module name inside the class file format should be plain string constant. regards, R?mi ----- Mail original ----- > De: "David M. Lloyd" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mardi 22 Novembre 2016 22:05:13 > Objet: Re: Proposal: #ModuleNameCharacters > On 11/22/2016 10:49 AM, mark.reinhold at oracle.com wrote: >> Proposal >> -------- >> >> Make no changes here. > > TL;DR: we can't accept this proposal as-is. Expounding more below. > >> Modules are a new construct of the Java programming language in the >> present design. In the source language they are hence identified by >> qualified names [2] in the same manner as the existing structural >> constructs, i.e., packages and classes. As such these names do allow >> some unusual characters, though not hyphens or slashes [3]. >> >> Module names in compiled module-declaration class files are recorded in >> `CONSTANT_Utf8_info` structures, and thus have fewer constraints. > > I believe that according to the JVM spec (until now anyway), this field > type by itself has no constraints at all, beyond being a valid > "modified" UTF-8 string. > >> They replace periods (`'.'`) with forward slashes (`'/'`), and disallow >> periods, semicolons (`';'`), and left square brackets (`'['`) [4]. > > These name manglings are really just plain weird in this context, and > are clearly an implementation artifact. How did we arrive at this > place? I feel like it springs from the long-maligned conflation of > module descriptors with class files. But modules are not types and > AFAICT these restrictions really make no sense from any other > perspective than one of implementation. > >> The `ModuleDescriptor` API can read class files that contain module names not >> expressible in the source language, though it cannot be used to construct >> instances with such names. > > This restriction seems arbitrary, but we can mitigate it I guess. > >> Artifacts that define modules with such names >> will be processed normally when placed on the module path. > > OK. Just to be clear though, the use cases that we are dealing with > would not involve a module path. > >> The present design is, then, consistent with the existing treatment of >> qualified names in the language, in class files, and in the Java SE API. > > I do not believe that any of these statements is sufficient to justify > the constraint... more below. > >> A different module system with a more-flexible naming scheme can easily >> refer to JPMS modules, per the agreed interoperation requirement [5]. >> The requirements do not mandate bidirectional interoperation, which for >> this issue would mean that JPMS modules must be able to refer to non-JPMS >> modules with non-JPMS names. > > This is also not sufficient to justify the constraint, though it does > help to explain the reasoning why the constraint existed in the first place. > >> To support that would add significant >> complexity to this specification and its implementations. > > I am sympathetic to this, but I think it needs more discussion, > particularly as the proposed complexities have not been explained. > > The overarching problem here for us is that the JPMS specifies a new > deployment model which is being put forth as a standard for future > development. However we can't accept a solution which will render > existing, otherwise satisfactory, alternative modular deployment models > unusable with this new model: for example OSGi, Java EE, or JBoss > Modules and its usages in various contexts. To do so would be, at a > minimum, a huge disservice to our users, and would require a large > investment to migrate to the new model: a lossy move, because in this > case we'd be losing features by definition. > > Thus one implicit requirement that we have is that we can successfully > evolve our deployment models into the JPMS and interoperate fully; else > we must choose between sentencing our own deployment models to death or > dismemberment, or else being unable to take advantage of the JPMS > capabilities (and in either case we would have no motivation to support > it, though for different reasons in each case). While we can > theoretically support some name constraints on our model (leaving Java > EE aside), the constraints we could support do not match the ones that > exist, and therefore this information doesn't really help much. > > But the other implicit requirement that we have is to ensure that it's > possible to adapt Java EE in some reasonable manner. It's my belief > that in order to do so we need to be able to create modules with names > that match the current constraints for Java EE module names (i.e. > effectively no constraint, just valid UTF-8). Name mangling to > accommodate this (which would be our only other option) is unnecessarily > user-unfriendly at best, and outright incompatible at worst. > > I think we should talk about the factors that are complicating this > issue and proceed from there... maybe there's something we can do to help. > > -- > - DML From forax at univ-mlv.fr Wed Nov 23 20:00:59 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 23 Nov 2016 21:00:59 +0100 (CET) Subject: Proposal: #ClassFileAccModule In-Reply-To: <20161122164835.6F47218943@eggemoggin.niobe.net> References: <20161122164835.6F47218943@eggemoggin.niobe.net> Message-ID: <1798661093.1641158.1479931259481.JavaMail.zimbra@u-pem.fr> fair enough, i agree. R?mi ----- Mail original ----- > De: "mark reinhold" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mardi 22 Novembre 2016 17:48:35 > Objet: Proposal: #ClassFileAccModule > Issue summary > ------------- > > #ClassFileAccModule --- The `ACC_MODULE` constant is currently > specified to have the value 0x8000. This is the last available bit > remaining across all of the various `access_flags` fields of a class > file, and thus should be reserved for some unspecified future purpose > where it may be useful to use the same value in all such fields. > Alternative candidates for `ACC_MODULE` include 0x0040 (overlaps with > `ACC_VOLATILE` and `ACC_BRIDGE`) and 0x0080 (`ACC_TRANSIENT` and > `ACC_VARARGS`). [1] > > Proposal > -------- > > Do not change the value of this constant. > > The high-order bits of the `access_flags` field have long been used to > characterize the unusual, non-class nature of some class files: > > - `ACC_ANNOTATION` (0x2000) indicates an annotation type, even though > `ACC_ANNOTATION` does not apply to fields or methods, and > > - `ACC_ENUM` (0x4000) indicates an `enum` type, even though `ACC_ENUM` > does not apply to methods. > > A binary module descriptor is a new kind of unusual class file, hence we > have done the obvious thing and allocated the high-order bit 0x8000 for > `ACC_MODULE`. > > We could try to be more clever here, but that would require attempting > to predict the future. There are still plenty of other ways to encode > potential future features; a value class could, e.g., be indicated by > 0x0100. In the worst case the `access_flags` field could simply be made > wider in some future version of the class-file specification. > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassFileAccModule From forax at univ-mlv.fr Wed Nov 23 20:03:25 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 23 Nov 2016 21:03:25 +0100 (CET) Subject: Proposal: #ClassFileModuleName In-Reply-To: <20161122164735.6899C1893F@eggemoggin.niobe.net> References: <20161122164735.6899C1893F@eggemoggin.niobe.net> Message-ID: <288915504.1641396.1479931405933.JavaMail.zimbra@u-pem.fr> I'm pleased with this proposal, if obviously, module name are NOT in internal form. regards, R?mi ----- Mail original ----- > De: "mark reinhold" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mardi 22 Novembre 2016 17:47:35 > Objet: Proposal: #ClassFileModuleName > Issue summary > ------------- > > #ClassFileModuleName --- The name of a module is not a simple UTF-8 > string but is, rather, derived from the value of the `this_class` > field of the `ClassFile` structure, which is awkward. [1] > > Proposal > -------- > > Revise the binary form of module declarations [2] as follows: > > - Add a `u2 module_name_index` field to the `Module` class-file > attribute, immediately preceding the existing `module_flags` field. > The value of this field will be the index of a `CONSTANT_Utf8_info` > structure in the constant pool that represents the module's name in > internal form, i.e., with period characters replaced by slashes [3]. > > - Rather than encode the module's name in the value of the `this_class` > field of the `ClassFile` structure, simply set this field to zero. > > (Side note: The prototype implementation does not store module names in > internal form, even though the draft specification [2] has long mandated > that. The implementation will be fixed.) > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassFileModuleName > [2] http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html > [3] http://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.2.1 From stef at epardaud.fr Thu Nov 24 09:22:46 2016 From: stef at epardaud.fr (Stephane Epardaud) Date: Thu, 24 Nov 2016 10:22:46 +0100 Subject: Proposal: #ModuleNameCharacters In-Reply-To: <2034282380.1641047.1479931198355.JavaMail.zimbra@u-pem.fr> References: <20161122164935.7633C18948@eggemoggin.niobe.net> <3c1891c8-b8c5-c1ba-ec87-34b7824fd7f7@redhat.com> <2034282380.1641047.1479931198355.JavaMail.zimbra@u-pem.fr> Message-ID: <5836B166.3060607@epardaud.fr> If you do, you'll have to find a way to encode the imported module names in `module-info.java`, though, so that you can import modules produced by other module systems/languages. What we have in Ceylon is that you can import foo.bar (Ceylon) modules as identifiers, or "foo-bar:gee" (Maven) modules as quoted strings. On 23/11/16 20:59, Remi Forax wrote: > I agree with David, > module names are just names so there should be encoded as 'UTF8' kind of constant with no restriction. > > Java (the language) module name format has more restrictions because javac has to compile module-info.java files, > but in the bytecode there is no need to encode module name as 'internal name'. > > When doing the JSR 292 spec, we (the EG) removed all the constraints on class name, method name, etc that could be removed, in order to ease the mapping of any languages on top of the VM. > If we want ease the mapping between any existing and future module system to JPMS, module name inside the class file format should be plain string constant. > > regards, > R?mi > > ----- Mail original ----- >> De: "David M. Lloyd" >> ?: jpms-spec-experts at openjdk.java.net >> Envoy?: Mardi 22 Novembre 2016 22:05:13 >> Objet: Re: Proposal: #ModuleNameCharacters >> On 11/22/2016 10:49 AM, mark.reinhold at oracle.com wrote: >>> Proposal >>> -------- >>> >>> Make no changes here. >> TL;DR: we can't accept this proposal as-is. Expounding more below. >> >>> Modules are a new construct of the Java programming language in the >>> present design. In the source language they are hence identified by >>> qualified names [2] in the same manner as the existing structural >>> constructs, i.e., packages and classes. As such these names do allow >>> some unusual characters, though not hyphens or slashes [3]. >>> >>> Module names in compiled module-declaration class files are recorded in >>> `CONSTANT_Utf8_info` structures, and thus have fewer constraints. >> I believe that according to the JVM spec (until now anyway), this field >> type by itself has no constraints at all, beyond being a valid >> "modified" UTF-8 string. >> >>> They replace periods (`'.'`) with forward slashes (`'/'`), and disallow >>> periods, semicolons (`';'`), and left square brackets (`'['`) [4]. >> These name manglings are really just plain weird in this context, and >> are clearly an implementation artifact. How did we arrive at this >> place? I feel like it springs from the long-maligned conflation of >> module descriptors with class files. But modules are not types and >> AFAICT these restrictions really make no sense from any other >> perspective than one of implementation. >> >>> The `ModuleDescriptor` API can read class files that contain module names not >>> expressible in the source language, though it cannot be used to construct >>> instances with such names. >> This restriction seems arbitrary, but we can mitigate it I guess. >> >>> Artifacts that define modules with such names >>> will be processed normally when placed on the module path. >> OK. Just to be clear though, the use cases that we are dealing with >> would not involve a module path. >> >>> The present design is, then, consistent with the existing treatment of >>> qualified names in the language, in class files, and in the Java SE API. >> I do not believe that any of these statements is sufficient to justify >> the constraint... more below. >> >>> A different module system with a more-flexible naming scheme can easily >>> refer to JPMS modules, per the agreed interoperation requirement [5]. >>> The requirements do not mandate bidirectional interoperation, which for >>> this issue would mean that JPMS modules must be able to refer to non-JPMS >>> modules with non-JPMS names. >> This is also not sufficient to justify the constraint, though it does >> help to explain the reasoning why the constraint existed in the first place. >> >>> To support that would add significant >>> complexity to this specification and its implementations. >> I am sympathetic to this, but I think it needs more discussion, >> particularly as the proposed complexities have not been explained. >> >> The overarching problem here for us is that the JPMS specifies a new >> deployment model which is being put forth as a standard for future >> development. However we can't accept a solution which will render >> existing, otherwise satisfactory, alternative modular deployment models >> unusable with this new model: for example OSGi, Java EE, or JBoss >> Modules and its usages in various contexts. To do so would be, at a >> minimum, a huge disservice to our users, and would require a large >> investment to migrate to the new model: a lossy move, because in this >> case we'd be losing features by definition. >> >> Thus one implicit requirement that we have is that we can successfully >> evolve our deployment models into the JPMS and interoperate fully; else >> we must choose between sentencing our own deployment models to death or >> dismemberment, or else being unable to take advantage of the JPMS >> capabilities (and in either case we would have no motivation to support >> it, though for different reasons in each case). While we can >> theoretically support some name constraints on our model (leaving Java >> EE aside), the constraints we could support do not match the ones that >> exist, and therefore this information doesn't really help much. >> >> But the other implicit requirement that we have is to ensure that it's >> possible to adapt Java EE in some reasonable manner. It's my belief >> that in order to do so we need to be able to create modules with names >> that match the current constraints for Java EE module names (i.e. >> effectively no constraint, just valid UTF-8). Name mangling to >> accommodate this (which would be our only other option) is unnecessarily >> user-unfriendly at best, and outright incompatible at worst. >> >> I think we should talk about the factors that are complicating this >> issue and proceed from there... maybe there's something we can do to help. >> >> -- >> - DML From forax at univ-mlv.fr Sat Nov 26 15:31:42 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Sat, 26 Nov 2016 16:31:42 +0100 (CET) Subject: Split packages Message-ID: <871414197.2537969.1480174302378.JavaMail.zimbra@u-pem.fr> I'm trying to be able to see mavencentral as a provider of jigsaw modules (automatic for now, so as a flat hierarchy), and obviously, i've to solve the problem of split packages. There is an obvious way to solve the split packages issue, merge the modules that have split packages into one bigger module. Writing a ModuleFinder that virtually group two modules together is easy, so i wonder if there is a simple way to a user to specify a kind of automatic grouping module that group together several automatic modules ? regards, R?mi From mark.reinhold at oracle.com Wed Nov 30 00:08:02 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 29 Nov 2016 16:08:02 -0800 (PST) Subject: Proposal: #ModuleNameCharacters In-Reply-To: <3c1891c8-b8c5-c1ba-ec87-34b7824fd7f7@redhat.com> References: <20161122164935.7633C18948@eggemoggin.niobe.net> <3c1891c8-b8c5-c1ba-ec87-34b7824fd7f7@redhat.com> Message-ID: <20161130000802.D65C81C6B0@eggemoggin.niobe.net> 2016/11/22 13:05:13 -0800, david.lloyd at redhat.com: > On 11/22/2016 10:49 AM, mark.reinhold at oracle.com wrote: >> Proposal >> -------- >> >> Make no changes here. > > TL;DR: we can't accept this proposal as-is. Expounding more below. > >> Modules are a new construct of the Java programming language in the >> present design. In the source language they are hence identified by >> qualified names [2] in the same manner as the existing structural >> constructs, i.e., packages and classes. As such these names do allow >> some unusual characters, though not hyphens or slashes [3]. >> >> Module names in compiled module-declaration class files are recorded in >> `CONSTANT_Utf8_info` structures, and thus have fewer constraints. > > I believe that according to the JVM spec (until now anyway), this field > type by itself has no constraints at all, beyond being a valid > "modified" UTF-8 string. It's true that a CONSTANT_Utf8_info can, as such, contain arbitrary UTF-8 text. The additional restrictions I listed are the same as those already in place for other kinds of qualified names, i.e., the names of packages and classes. A module name is, after all, just another kind of qualified name in the present design. >> They replace periods (`'.'`) with forward slashes (`'/'`), and disallow >> periods, semicolons (`';'`), and left square brackets (`'['`) [4]. > > These name manglings are really just plain weird in this context, and > are clearly an implementation artifact. How did we arrive at this > place? I feel like it springs from the long-maligned conflation of > module descriptors with class files. But modules are not types and > AFAICT these restrictions really make no sense from any other > perspective than one of implementation. These restrictions make perfect sense from the perspective of tools that manipulate class files, which are likely already to assume that these restrictions are observed uniformly for all kinds of qualified names. I'm reluctant to violate that expectation without good reason. These restrictions have, moreover, ensured a useful degree of freedom for the evolution of the platform over the last twenty years. I'm reluctant to give that up without good reason. > ... > >> The present design is, then, consistent with the existing treatment of >> qualified names in the language, in class files, and in the Java SE API. > > I do not believe that any of these statements is sufficient to justify > the constraint... more below. > >> A different module system with a more-flexible naming scheme can easily >> refer to JPMS modules, per the agreed interoperation requirement [5]. >> The requirements do not mandate bidirectional interoperation, which for >> this issue would mean that JPMS modules must be able to refer to non-JPMS >> modules with non-JPMS names. > > This is also not sufficient to justify the constraint, though it does > help to explain the reasoning why the constraint existed in the first place. There is no agreed requirement for bidirectional interoperation, nor any other agreed requirement that mandates that module names be arbitrary strings. There is thus no need, per the requirements, to support such module names. In the absence of a requirement to support arbitrary module names I've chosen in the present design to be consistent with the other kinds of qualified names already used in the language. (I acknowledge that you don't think that modules should be a language construct in the first place, but that's not a decision that I intend to revisit.) This will be the least surprising choice for the vast majority of developers who will use this module system. >> To support that would add significant >> complexity to this specification and its implementations. > > I am sympathetic to this, but I think it needs more discussion, > particularly as the proposed complexities have not been explained. I'm concerned mostly about the complexity of the language specification, which affects all users of this module system. If we change the language of module declarations to allow arbitrary module names, possibly via some sort of quoting scheme, then that's something that every developer would have to understand even though it would most likely be of benefit to few. I'm concerned also, though to a lesser degree, about the complexity of the class-file specification (i.e., the JVMS), the long-term evolvability of that specification, and the complexity of code that reads and writes class files, though the latter is second-order since it mostly affects maintainers of IDEs, compilers, and other kinds of tools rather than developers in general. If there's a compelling reason to lift the usual restrictions on the representation of qualified names in class files, at least in the case of module names, then I'd like to hear it. > ... > > > But the other implicit requirement that we have is to ensure that it's > possible to adapt Java EE in some reasonable manner. It's my belief > that in order to do so we need to be able to create modules with names > that match the current constraints for Java EE module names (i.e. > effectively no constraint, just valid UTF-8). Name mangling to > accommodate this (which would be our only other option) is unnecessarily > user-unfriendly at best, and outright incompatible at worst. I don't expect Java EE modules to map directly to Java SE modules, nor do any of the Java EE spec leads with whom I've discussed this issue. EE modules and SE modules are completely different kinds of things. - Mark From mark.reinhold at oracle.com Wed Nov 30 00:09:02 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 29 Nov 2016 16:09:02 -0800 (PST) Subject: Discussion: #MutableConfigurations, #LazyConfigurationAndInstantiation, #CyclicDependences, & #DiscardableModules In-Reply-To: References: <20161118163009.8C54A16AC5@eggemoggin.niobe.net> <8e5bd410-9737-db44-6c2f-4289ad4c8db2@redhat.com> <20161121135201.534880609@eggemoggin.niobe.net> Message-ID: <20161130000902.DB5A31C6B3@eggemoggin.niobe.net> 2016/11/22 13:19:19 -0800, david.lloyd at redhat.com: > On 11/21/2016 03:52 PM, mark.reinhold at oracle.com wrote: >> ... >> >> If you're using an approach similar to what I suggested for OSGi [1] >> then you should be able to implement cycles amongst JBoss modules by >> inserting the necessary readability edges after each module is >> instantiated in its own layer. This will be much easier with the new >> API proposed for #ReadabilityAddedByLayerCreator [2]. > > I asked earlier whether this allows the class resolver (What's a "class resolver" in this context?) > to resolve > against modules with a lately-added readability edge as you describe > here, and it sounded like the answer was "no"; did I miss something? No, I don't think so. Readability edges added at run time only affect run-time access-control checks; they don't affect later module-resolution operations. > If > my understanding was right, then this doesn't allow for circularity at > the class level where we need it. The JPMS resolver does not allow cycles amongst modules; this has long been the case. (Circularity amongst classes is allowed, as it must be.) If you want to allow cycles amongst your own modules then you can resolve them yourself and add whatever cycle-inducing readability edges you need. That is, as I understand it, how Watson's OSGi embedding works. - Mark From mark.reinhold at oracle.com Wed Nov 30 00:10:02 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 29 Nov 2016 16:10:02 -0800 (PST) Subject: Proposal: #NonHierarchicalLayers In-Reply-To: <0a1e41c6-9d36-ad5a-5ad0-7445e7ae3883@redhat.com> References: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> <0f23ba26-9e93-a62b-b537-be0d5a91b895@redhat.com> <20161121135638.309938232@eggemoggin.niobe.net> <0a1e41c6-9d36-ad5a-5ad0-7445e7ae3883@redhat.com> Message-ID: <20161130001002.E23C91C6BC@eggemoggin.niobe.net> 2016/11/22 13:23:34 -0800, david.lloyd at redhat.com: > On 11/21/2016 03:56 PM, mark.reinhold at oracle.com wrote: >> ... >> >> Layers don't have names, but if you're using the approach I suggested >> for OSGi [1] then it shouldn't matter, since every JBoss module will be >> in its own layer. (If you really do need to give layers separate names >> then you could do so in a `WeakHashMap` that you maintain on the side, >> and have your resolver refer to that.) > > No, I was just suggesting a solution to the problem I see. > > To degenerate the problem to the simplest case: I may have to layers, A > and B, which are unrelated and both contain a module X. I may need the > module X in one layer to have a dependency on the module X in another > layer. Unless there's new code I haven't seen, each dependency is > expressed by name only, therefore this is impossible. Whereas in our > system, a dependency is a tuple of (module loader, module name), > allowing arbitrary weaving of modules from different name spaces with no > problem. > > I don't think we need it to work the exact same way, but we need some > equivalent. If this is how you need resolution to work for your modules then can't you write your own resolver to make it so, using whatever additional (weak) data structures you need to associate names with layers, class loaders, etc., in whatever manner you require? - Mark From mark.reinhold at oracle.com Wed Nov 30 00:11:02 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 29 Nov 2016 16:11:02 -0800 (PST) Subject: Proposal: #ModuleNameCharacters In-Reply-To: <2034282380.1641047.1479931198355.JavaMail.zimbra@u-pem.fr> References: <20161122164935.7633C18948@eggemoggin.niobe.net> <3c1891c8-b8c5-c1ba-ec87-34b7824fd7f7@redhat.com> <2034282380.1641047.1479931198355.JavaMail.zimbra@u-pem.fr> Message-ID: <20161130001102.E7C5C1C6C1@eggemoggin.niobe.net> 2016/11/23 11:59:58 -0800, forax at univ-mlv.fr: > I agree with David, module names are just names so there should be > encoded as 'UTF8' kind of constant with no restriction. > > Java (the language) module name format has more restrictions because > javac has to compile module-info.java files, but in the bytecode there > is no need to encode module name as 'internal name'. > > When doing the JSR 292 spec, we (the EG) removed all the constraints > on class name, method name, etc that could be removed, in order to > ease the mapping of any languages on top of the VM. Most, if not all, such constraints were removed long before JSR 292. For those that remain, John Rose proposed a fairly straightforward name-mangling scheme [1]. > If we want ease > the mapping between any existing and future module system to JPMS, > module name inside the class file format should be plain string > constant. As I wrote in my reply to David, I'm open to lifting the traditional restrictions on the class-file representation of qualified names in the case of module names. Given the weight of tradition and the past value of the existing constraints, however, I'd like to have a more compelling reason than "some future hypothetical module system might need this flexibility". In trying to think about the future I do wonder if, today, we should reserve a character or two just in case we discover five or ten years from now that we need to add more structure to module names. Should we set aside `:`, or perhaps some other character, just in case? - Mark [1] https://blogs.oracle.com/jrose/entry/symbolic_freedom_in_the_vm From mark.reinhold at oracle.com Wed Nov 30 00:23:05 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 29 Nov 2016 16:23:05 -0800 Subject: Closing out more open issues In-Reply-To: <20161118162821.56770200@eggemoggin.niobe.net> References: <20161118162821.56770200@eggemoggin.niobe.net> Message-ID: <20161129162305.98358300@eggemoggin.niobe.net> 2016/11/18 16:28:21 -0800, mark.reinhold at oracle.com: > Proposals for the following issues have been available for evaluation > and experimentation for some time [1]: > > #AddExportsInManifest > #ClassFilesAsResources > #ClassLoaderNames > #ResourceEncapsulation > #ServiceLoaderEnhancements > #VersionsInModuleNames > > ... > > If no EG member objects by this time next Friday, 25 November, then > I'll close these issues out. Hearing no objections, I've closed these issues as resolved [1]. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ From david.lloyd at redhat.com Wed Nov 30 02:47:33 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 29 Nov 2016 20:47:33 -0600 Subject: Proposal: #ModuleNameCharacters In-Reply-To: <20161130000802.D65C81C6B0@eggemoggin.niobe.net> References: <20161122164935.7633C18948@eggemoggin.niobe.net> <3c1891c8-b8c5-c1ba-ec87-34b7824fd7f7@redhat.com> <20161130000802.D65C81C6B0@eggemoggin.niobe.net> Message-ID: <19808667-7501-fc47-009d-b245e3a099ce@redhat.com> On 11/29/2016 06:08 PM, mark.reinhold at oracle.com wrote: > 2016/11/22 13:05:13 -0800, david.lloyd at redhat.com: >> On 11/22/2016 10:49 AM, mark.reinhold at oracle.com wrote: >>> Proposal >>> -------- >>> >>> Make no changes here. >> >> TL;DR: we can't accept this proposal as-is. Expounding more below. >> >>> Modules are a new construct of the Java programming language in the >>> present design. In the source language they are hence identified by >>> qualified names [2] in the same manner as the existing structural >>> constructs, i.e., packages and classes. As such these names do allow >>> some unusual characters, though not hyphens or slashes [3]. >>> >>> Module names in compiled module-declaration class files are recorded in >>> `CONSTANT_Utf8_info` structures, and thus have fewer constraints. >> >> I believe that according to the JVM spec (until now anyway), this field >> type by itself has no constraints at all, beyond being a valid >> "modified" UTF-8 string. > > It's true that a CONSTANT_Utf8_info can, as such, contain arbitrary UTF-8 > text. The additional restrictions I listed are the same as those already > in place for other kinds of qualified names, i.e., the names of packages > and classes. A module name is, after all, just another kind of qualified > name in the present design. Right, that's the implementation, but that is a problem for us. >>> They replace periods (`'.'`) with forward slashes (`'/'`), and disallow >>> periods, semicolons (`';'`), and left square brackets (`'['`) [4]. >> >> These name manglings are really just plain weird in this context, and >> are clearly an implementation artifact. How did we arrive at this >> place? I feel like it springs from the long-maligned conflation of >> module descriptors with class files. But modules are not types and >> AFAICT these restrictions really make no sense from any other >> perspective than one of implementation. > > These restrictions make perfect sense from the perspective of tools that > manipulate class files, which are likely already to assume that these > restrictions are observed uniformly for all kinds of qualified names. ...but not for all CONSTANT_Utf8_info fields, obviously. For example, String constant values use CONSTANT_Utf8_info without requiring name mangling. > I'm reluctant to violate that expectation without good reason. AFAICT nobody has this expectation. Who would expect that a module name can't have left square brackets? What possible reason could that serve? > These restrictions have, moreover, ensured a useful degree of freedom for > the evolution of the platform over the last twenty years. I'm reluctant > to give that up without good reason. For types, sure; but this is a module name, not a type or any part of a type. I just don't see the equivalency. >>> The present design is, then, consistent with the existing treatment of >>> qualified names in the language, in class files, and in the Java SE API. >> >> I do not believe that any of these statements is sufficient to justify >> the constraint... more below. >> >>> A different module system with a more-flexible naming scheme can easily >>> refer to JPMS modules, per the agreed interoperation requirement [5]. >>> The requirements do not mandate bidirectional interoperation, which for >>> this issue would mean that JPMS modules must be able to refer to non-JPMS >>> modules with non-JPMS names. >> >> This is also not sufficient to justify the constraint, though it does >> help to explain the reasoning why the constraint existed in the first place. > > There is no agreed requirement for bidirectional interoperation, nor any > other agreed requirement that mandates that module names be arbitrary > strings. There is thus no need, per the requirements, to support such > module names. Nevertheless, we must do so, and this particular enhancement does not seem to be a big "ask". > In the absence of a requirement to support arbitrary module names I've > chosen in the present design to be consistent with the other kinds of > qualified names already used in the language. (I acknowledge that you > don't think that modules should be a language construct in the first > place, but that's not a decision that I intend to revisit.) This will > be the least surprising choice for the vast majority of developers who > will use this module system. Again, nobody will have this expectation. >>> To support that would add significant >>> complexity to this specification and its implementations. >> >> I am sympathetic to this, but I think it needs more discussion, >> particularly as the proposed complexities have not been explained. > > I'm concerned mostly about the complexity of the language specification, > which affects all users of this module system. If we change the language > of module declarations to allow arbitrary module names, possibly via some > sort of quoting scheme, then that's something that every developer would > have to understand even though it would most likely be of benefit to few. You don't need a quoting scheme to allow arbitrary names - you just allow them - and I don't see how this complicates the language specification in any way: in fact it should simplify it. There's just no reason to use "internal form". > I'm concerned also, though to a lesser degree, about the complexity of > the class-file specification (i.e., the JVMS), the long-term evolvability > of that specification, and the complexity of code that reads and writes > class files, though the latter is second-order since it mostly affects > maintainers of IDEs, compilers, and other kinds of tools rather than > developers in general. If there's a compelling reason to lift the usual > restrictions on the representation of qualified names in class files, at > least in the case of module names, then I'd like to hear it. I'm just really confused at this line of justification. Making module names be a qualified name is the very thing that I'm arguing against. Don't make any change to the restrictions on qualified names, and don't make module names be qualified names; that's all there is to it. >> But the other implicit requirement that we have is to ensure that it's >> possible to adapt Java EE in some reasonable manner. It's my belief >> that in order to do so we need to be able to create modules with names >> that match the current constraints for Java EE module names (i.e. >> effectively no constraint, just valid UTF-8). Name mangling to >> accommodate this (which would be our only other option) is unnecessarily >> user-unfriendly at best, and outright incompatible at worst. > > I don't expect Java EE modules to map directly to Java SE modules, nor > do any of the Java EE spec leads with whom I've discussed this issue. > EE modules and SE modules are completely different kinds of things. On March 11 of this year (and other occasions) I specifically asked exactly that, and you said [1] "Of course we have that expectation -- that's why the requirements include an entire section on dynamic configuration". Did I misunderstand you then or now, or has something changed? If something has changed, can we get agreement from this and the Java EE expert groups that Java EE will not make this a requirement until/unless both EGs agree to do so at some later time (for example, pending updates in Java 10)? [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-March/000251.html -- - DML From david.lloyd at redhat.com Wed Nov 30 02:47:59 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 29 Nov 2016 20:47:59 -0600 Subject: Discussion: #MutableConfigurations, #LazyConfigurationAndInstantiation, #CyclicDependences, & #DiscardableModules In-Reply-To: <20161130000902.DB5A31C6B3@eggemoggin.niobe.net> References: <20161118163009.8C54A16AC5@eggemoggin.niobe.net> <8e5bd410-9737-db44-6c2f-4289ad4c8db2@redhat.com> <20161121135201.534880609@eggemoggin.niobe.net> <20161130000902.DB5A31C6B3@eggemoggin.niobe.net> Message-ID: <7bb95e6d-a127-ea4f-7158-d9f062e8aed3@redhat.com> On 11/29/2016 06:09 PM, mark.reinhold at oracle.com wrote: > 2016/11/22 13:19:19 -0800, david.lloyd at redhat.com: >> On 11/21/2016 03:52 PM, mark.reinhold at oracle.com wrote: >>> ... >>> >>> If you're using an approach similar to what I suggested for OSGi [1] >>> then you should be able to implement cycles amongst JBoss modules by >>> inserting the necessary readability edges after each module is >>> instantiated in its own layer. This will be much easier with the new >>> API proposed for #ReadabilityAddedByLayerCreator [2]. >> >> I asked earlier whether this allows the class resolver > > (What's a "class resolver" in this context?) The part of the class loader that resolves references between classes and performs linkage. There seems to be no non-overloaded term for this, anymore. >> to resolve >> against modules with a lately-added readability edge as you describe >> here, and it sounded like the answer was "no"; did I miss something? > > No, I don't think so. Readability edges added at run time only affect > run-time access-control checks; they don't affect later module-resolution > operations. > >> If >> my understanding was right, then this doesn't allow for circularity at >> the class level where we need it. > > The JPMS resolver does not allow cycles amongst modules; this has long > been the case. (Circularity amongst classes is allowed, as it must be.) Right, and therefore we have come full circle to #CyclicDependences. > If you want to allow cycles amongst your own modules then you can resolve > them yourself and add whatever cycle-inducing readability edges you need. > That is, as I understand it, how Watson's OSGi embedding works. But it fails when the actual classes have direct references to one another. It's not sufficient. The issue document says "easier", and talks about philosophical points, but here in the mud the reality is that it's just easier to allow cycles (in fact we made this change over 5 years ago). Debugging an unexpected run time cycle is a major pain, whereas allowing cycles cost us nothing and instantly alleviated a number of practical difficulties (everything *just works*). The grim reality is that a module dependencies simply do not always form DAGs at run time in the real world, especially once you have hundreds or thousands of interconnected modules, any more than classes do - and it's an unreasonable restriction to try to force it, for many of the same reasons that such a restriction would be unreasonable for classes. -- - DML From david.lloyd at redhat.com Wed Nov 30 02:50:11 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 29 Nov 2016 20:50:11 -0600 Subject: Proposal: #ModuleNameCharacters In-Reply-To: <20161130001102.E7C5C1C6C1@eggemoggin.niobe.net> References: <20161122164935.7633C18948@eggemoggin.niobe.net> <3c1891c8-b8c5-c1ba-ec87-34b7824fd7f7@redhat.com> <2034282380.1641047.1479931198355.JavaMail.zimbra@u-pem.fr> <20161130001102.E7C5C1C6C1@eggemoggin.niobe.net> Message-ID: <69a96954-009a-125d-98dd-0f3cb341e59d@redhat.com> On 11/29/2016 06:11 PM, mark.reinhold at oracle.com wrote: > 2016/11/23 11:59:58 -0800, forax at univ-mlv.fr: >> I agree with David, module names are just names so there should be >> encoded as 'UTF8' kind of constant with no restriction. >> >> Java (the language) module name format has more restrictions because >> javac has to compile module-info.java files, but in the bytecode there >> is no need to encode module name as 'internal name'. >> >> When doing the JSR 292 spec, we (the EG) removed all the constraints >> on class name, method name, etc that could be removed, in order to >> ease the mapping of any languages on top of the VM. > > Most, if not all, such constraints were removed long before JSR 292. > For those that remain, John Rose proposed a fairly straightforward > name-mangling scheme [1]. > >> If we want ease >> the mapping between any existing and future module system to JPMS, >> module name inside the class file format should be plain string >> constant. > > As I wrote in my reply to David, I'm open to lifting the traditional > restrictions on the class-file representation of qualified names in the > case of module names. Given the weight of tradition and the past value > of the existing constraints, however, I'd like to have a more compelling > reason than "some future hypothetical module system might need this > flexibility". The key point is exactly that a present, concrete module system does need this flexibility; ours, specifically. And this is such an easy change. > In trying to think about the future I do wonder if, today, we should > reserve a character or two just in case we discover five or ten years > from now that we need to add more structure to module names. Should > we set aside `:`, or perhaps some other character, just in case? We have had a module system with no character restrictions for many years now and never looked back. It adds flexibility as every module loader can establish acceptable name rules based on its policy, which afforded us a great deal of interoperability with all manner of system. You can still have reserved characters if that's what is appropriate to your module resolver; if a name is invalid for that resolver, the resolver can simply return no match. -- - DML From david.lloyd at redhat.com Wed Nov 30 02:51:22 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 29 Nov 2016 20:51:22 -0600 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: <20161130001002.E23C91C6BC@eggemoggin.niobe.net> References: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> <0f23ba26-9e93-a62b-b537-be0d5a91b895@redhat.com> <20161121135638.309938232@eggemoggin.niobe.net> <0a1e41c6-9d36-ad5a-5ad0-7445e7ae3883@redhat.com> <20161130001002.E23C91C6BC@eggemoggin.niobe.net> Message-ID: <2c7a0dbb-3666-dd7a-3260-e8bc4b8c10f2@redhat.com> On 11/29/2016 06:10 PM, mark.reinhold at oracle.com wrote: > 2016/11/22 13:23:34 -0800, david.lloyd at redhat.com: >> On 11/21/2016 03:56 PM, mark.reinhold at oracle.com wrote: >>> ... >>> >>> Layers don't have names, but if you're using the approach I suggested >>> for OSGi [1] then it shouldn't matter, since every JBoss module will be >>> in its own layer. (If you really do need to give layers separate names >>> then you could do so in a `WeakHashMap` that you maintain on the side, >>> and have your resolver refer to that.) >> >> No, I was just suggesting a solution to the problem I see. >> >> To degenerate the problem to the simplest case: I may have to layers, A >> and B, which are unrelated and both contain a module X. I may need the >> module X in one layer to have a dependency on the module X in another >> layer. Unless there's new code I haven't seen, each dependency is >> expressed by name only, therefore this is impossible. Whereas in our >> system, a dependency is a tuple of (module loader, module name), >> allowing arbitrary weaving of modules from different name spaces with no >> problem. >> >> I don't think we need it to work the exact same way, but we need some >> equivalent. > > If this is how you need resolution to work for your modules then can't > you write your own resolver to make it so, using whatever additional > (weak) data structures you need to associate names with layers, class > loaders, etc., in whatever manner you require? No, because the input to the resolver is just a name; there's no other context to say which layer the module comes from, without some kind of name mangling scheme which raises more problems (we ran into exactly this issue more than six years ago). In other words I can require a module named "X", but I can't say that I require the module "X" from this other layer/namespace; rather I have to flatten out all my namespaces in a way that can be mutually resolvable, which means that I must ultimately constrain user module names in some weird way. This is not so good when modules refer to one another by name programmatically, nor in diagnostic output. -- - DML From forax at univ-mlv.fr Wed Nov 30 23:19:35 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 1 Dec 2016 00:19:35 +0100 (CET) Subject: Proposal: #ClassFileModuleName In-Reply-To: <288915504.1641396.1479931405933.JavaMail.zimbra@u-pem.fr> References: <20161122164735.6899C1893F@eggemoggin.niobe.net> <288915504.1641396.1479931405933.JavaMail.zimbra@u-pem.fr> Message-ID: <2034784044.1251203.1480547975649.JavaMail.zimbra@u-pem.fr> Hi Mark, I've just finished to update ASM6 to use the new classfile format. Practically, setting this_class to zero create several NPEs in the the visitors provided with ASM because a lot of them try to use indexOf on the name of the class (to find the package name). Given that the classfile for a package-info is named "foo/bar/package-info", i think it's a good idea to use "module-info" instead of null as class name for a module-info.class, it will help at least my users . regards, R?mi ----- Mail original ----- > De: "Remi Forax" > ?: "mark reinhold" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Mercredi 23 Novembre 2016 21:03:25 > Objet: Re: Proposal: #ClassFileModuleName > I'm pleased with this proposal, > if obviously, module name are NOT in internal form. > > regards, > R?mi > > ----- Mail original ----- >> De: "mark reinhold" >> ?: jpms-spec-experts at openjdk.java.net >> Envoy?: Mardi 22 Novembre 2016 17:47:35 >> Objet: Proposal: #ClassFileModuleName > >> Issue summary >> ------------- >> >> #ClassFileModuleName --- The name of a module is not a simple UTF-8 >> string but is, rather, derived from the value of the `this_class` >> field of the `ClassFile` structure, which is awkward. [1] >> >> Proposal >> -------- >> >> Revise the binary form of module declarations [2] as follows: >> >> - Add a `u2 module_name_index` field to the `Module` class-file >> attribute, immediately preceding the existing `module_flags` field. >> The value of this field will be the index of a `CONSTANT_Utf8_info` >> structure in the constant pool that represents the module's name in >> internal form, i.e., with period characters replaced by slashes [3]. >> >> - Rather than encode the module's name in the value of the `this_class` >> field of the `ClassFile` structure, simply set this field to zero. >> >> (Side note: The prototype implementation does not store module names in >> internal form, even though the draft specification [2] has long mandated >> that. The implementation will be fixed.) >> >> >> [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassFileModuleName >> [2] http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html > > [3] http://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.2.1