Updated conformance text for Java SE 9

[Volker Simonis]

Hi Mark,

the overall wording sounds good to me. However I wonder if the
"upgradeable" attribute of a module shouldn't be made more explicit?

The current specifications seems to be very vague, in the sense that
it uses expressions like "by example"

    "A Java SE module is upgradeable if, and only if, it defines an API
    that is subject to replacement.  Only a small number of Java SE
    modules fall into this category.  Examples are.."

or "most":

    "A Java SE module is non-upgradeable if it is essential to the
    integrity of the Platform.  Most Java SE modules fall into this
    category.."
     => what's the definition of ESSENTIAL here?

or "typically, generally":
    "Java SE modules typically indicate in their specifications whether
    they are upgradeable.  Non-Java SE modules generally need no
    such indication..."

So I wonder if there's a fixed (i.e. mandated by the standard) set of
modules which HAVE TO BE upgradable and which AREN'T ALLOWED to be
upgradable? I also don't know if this has been discussed before, but
wouldn't it be better to make the "upgradeable" attribute explicit by
either encoding it into the module info of a module or by introducing
a new annotation on the module (equivalent to "@Deprecated"
annotation)?

Finally, what are the rules for upgradeable modules with respect to
requires/requires transitive? Does an upgradeable module need to have
the exactly same dependencies like the module it replaces?

And what about the "exports" of an upgradeable module? Can it extend
the exports of the module it replaces? And if not, should this be
checked by the implementation?

You mention that "an Implementation is not required to check" if an
upgradeable module really "exports APIs that are newer than those
exported by the version in the Implementation". But can an upgradeable
module really export a changed or completely different API and isn't
the Implementation required to check that? If the answer is yes, an
upgradeable module could easily re-export (e.g. by wrapping) internal
packages which the Implementation qualifiedly exports to the
Implementation module. (I've just checked that with the current
Reference Implementation it is for example possible to put a custom
version of jdk.jshell onto the --upgrade-module-path which re-exports
functionality from jdk.internal.misc.Unsafe).

The new spec also mentions that "a non-Java SE module is upgradeable
unless the Implementation deems otherwise". As far as I can an see in
the current Reference Implementation, this is not mentioned for any of
the non-Java SE modules. Nevertheless most of them are not really
upgradeable because of the hashing of dependent modules. So it seems
that in the current Reference Implementation the fact if a module is
considered "upgradable" is decided by the regexp given to the jmod
command to exclude modules from being hashed (e.g.
(?!java.activation|java.corba|java.se.ee|java.transaction|java.xml.bind|java.xml.ws|java.xml.ws.annotation|jdk.rmic|jdk.xml.bind|jdk.xml.ws)
for java.base). Again, as previously mentioned, I think an explicit
tagging would be better here.

I understand that if you have control over the command line, you can
easily circumvent any safeguards enforced by the VM so I don't want to
get into a security discussion here. I just try to understand what are
the exact rules for upgradeable modules and what (if any) checks the
specification requires from the Implementation.

Regards,
Volker


On Fri, Jul 14, 2017 at 5:19 PM,  <mark.reinhold at oracle.com> wrote:
> On a related note, here is some additional text to explain how
> upgradeable modules can be used to replace selected APIs that,
> in past releases, could be replaced via the Endorsed Standards
> Override Mechanism.
>
> Comments welcome.
>
> - Mark
>
> ----
>
> Upgradeable modules
> -------------------
>
> This Specification permits the user to replace certain APIs in an
> Implementation of the Java SE Platform with newer versions, without
> making any changes to that Implementation.
>
> Two kinds of APIs are subject to replacement.  First, there are APIs in
> the Java SE Platform that correspond to standards maintained outside of
> the Java Community Process, such as CORBA.  Second, there are APIs in the
> Java SE Platform that evolve with the Platform via the Java Community
> Process, but which are intended to be usable on their own, i.e.,
> _standalone_, on versions of the Platform other than the version where
> they were developed.
>
> Prior to Java SE 9, the [Endorsed Standards Override Mechanism][esom]
> allowed the replacement of individual packages in APIs maintained outside
> of the JCP, but allowed only a complete replacement of APIs evolved via
> the JCP.  Java SE 9 takes advantage of the modular structure of the Java
> SE Platform to allow the replacement of both kinds of APIs, and their
> implementations, on a uniform module-by-module basis.  Specifically, Java
> SE 9 categorizes each module in an Implementation as either _upgradeable_
> or _non-upgradeable_, and allows an upgradeable module to be replaced by
> the user without modifying or re-building the Implementation.  (The
> Endorsed Standards Override Mechanism was deprecated in a Maintenance
> Release of Java SE 8 and is not present in Java SE 9.)
>
> An Implementation of the Java SE 9 Platform must provide a means to
> replace those modules which are categorized as upgradeable, and must not
> provide a means to replace those modules which are categorized as
> non-upgradeable.  The rules for categorizing a module are as follows:
>
>   - A Java SE module is upgradeable if, and only if, it defines an API
>     that is subject to replacement.  Only a small number of Java SE
>     modules fall into this category.  Examples are the `java.corba`
>     module, which corresponds to a standard maintained outside of the
>     Java Community Process, and the `java.transaction` and
>     `java.xml.bind` modules, which are intended to be usable standalone.
>
>   - A Java SE module is non-upgradeable if it is essential to the
>     integrity of the Platform.  Most Java SE modules fall into this
>     category, such as `java.base`, `java.xml`, and `java.management`.
>
>   - A non-Java SE module is upgradeable unless the Implementation deems
>     otherwise.
>
> In order to replace an upgradeable module, the user provides an alternate
> version which the Implementation observes in preference to the version
> included in the Implementation.  The alternate version typically exports
> APIs that are newer than those exported by the version in the
> Implementation, although an Implementation is not required to check this.
>
> (The Reference Implementation provides this capability via the
> `--upgrade-module-path` command-line option.)
>
> Java SE modules typically indicate in their specifications whether they
> are upgradeable.  Non-Java SE modules generally need no such indication.
>
> In Java SE 9, most of the Java SE modules which are upgradeable are
> [deprecated for removal][#s10].
>
>
> [s10]: XXX #s10 (http://cr.openjdk.java.net/~iris/se/9/java-se-9-pr-spec/#s10)
> [esom]: http://docs.oracle.com/javase/8/docs/technotes/guides/extensions/index.html