Updated conformance text for Java SE 9

[Volker Simonis]

On Sat, Jul 22, 2017 at 1:46 AM,  <mark.reinhold at oracle.com> wrote:
> 2017/7/21 7:15:44 -0700, volker.simonis at gmail.com:
>> On Tue, Jul 18, 2017 at 12:33 AM, mark.reinhold at oracle.com wrote:
>>> 2017/7/17 8:21:26 -0700, volker.simonis at gmail.com:
>>> ...
>>>>
>>>>                                           If the answer is yes, an
>>>> upgradeable module could easily re-export (e.g. by wrapping) internal
>>>> packages which the Implementation qualifiedly exports to the
>>>> Implementation module. (I've just checked that with the current
>>>> Reference Implementation it is for example possible to put a custom
>>>> version of jdk.jshell onto the --upgrade-module-path which re-exports
>>>> functionality from jdk.internal.misc.Unsafe).
>>>
>>> You can do that if you're using an exploded JDK 9 image.  It shouldn't
>>> work with a proper linked image -- if it does, that's bug.
>>
>> You only can' t do it with a properly linked image because the
>> reference Implementation hashes the modules to which qualified exports
>> exist. But this hashing is an implementation detail of the reference
>> Implementation and not a "standard" feature (which I personally
>> regret). Even this hashing "safe-belt" can be easily disabled by a
>> corresponding "--patch-module jdk.jshell=XXX" option where 'XXX' can
>> be a non-existing directory. But that's probably something for the
>> jigsaw mailing list.
>
> FYI, the bug that enabled that hack was fixed not too long ago.
>

You're right. I obviously tested with an older version. Just for the
records: the issue was fixed under JDK-8177844 in jigsaw/jake and
integrated into jdk9 under JDK-8178380. Funny to see that others have
already detected and fixed this :)

>> So back to the specification. If we have a fixed list of standard
>> modules which are upgradeable, I don't understand why all the other,
>> non-SE modules are upgradeable by default (i.e. "A non-Java SE module
>> is upgradeable unless the Implementation deems otherwise"). Why not
>> define it the other way round that non-SE modules aren't upgradeable
>> until the Implementation deems otherwise? Do you really think
>> upgradeable non-standard modules will be useful? If yes, in which
>> context?
>
> Yes: Consider a custom linked image that contains some library and
> application modules.  You might want to override some of those modules
> in production, say, without having to (and maybe without being able to)
> re-link the image.
>
> I'll add some text to the specification to suggest this use case.
>

So you say that not only standard or platform modules can be
upgradeable, but also application modules which are linked into a
custom image? That's interesting - I haven't thought about that use
case. Please see my further comments below.

>> And there's another problem with the upgradeable modules. Because they
>> are upgradeable they obviously can't be hashed. So if somebody builds
>> a custom image without an upgradeable module, a user of that image
>> could easily provide his own module with the same name like the
>> upgradeable module and use all of the qualified exports from the
>> platform modules to the original version of the upgradeable module.
>> Don't you think that this could compromise the integrity of the
>> customized jlinked image?
>
> Yes, it certainly could!  That's why we've taken care in the RI to make
> sure that no interesting packages are exported to upgradeable modules.
> In fact, the only packages that are exported with qualification to
> upgradeable modules are from other upgradeable modules.
>

So if you add the section about overriding user modules from a custom
linked image which you've proposed above you should also add a warning
that creators of such images should be careful with the qualified
exports to such modules because they can be upgraded by a user and
compromise the integrity of their application. That's probably not as
dangerous as compromising the integrity of the platform, but that may
be not what users expect from modularization and custom images.

>>>> The new spec also mentions that "a non-Java SE module is upgradeable
>>>> unless the Implementation deems otherwise". As far as I can an see in
>>>> the current Reference Implementation, this is not mentioned for any of
>>>> the non-Java SE modules. Nevertheless most of them are not really
>>>> upgradeable because of the hashing of dependent modules. So it seems
>>>> that in the current Reference Implementation the fact if a module is
>>>> considered "upgradable" is decided by the regexp given to the jmod
>>>> command to exclude modules from being hashed (e.g.
>>>> (?!java.activation|java.corba|java.se.ee|java.transaction|java.xml.bind|java.xml.ws|java.xml.ws.annotation|jdk.rmic|jdk.xml.bind|jdk.xml.ws)
>>>> for java.base). Again, as previously mentioned, I think an explicit
>>>> tagging would be better here.
>>>
>>> I suggest that the explicit specification of upgradeability is adequate.
>>
>> Yes, but as far as I understood, that will be done only for the SE
>> modules and not for the other platform modules.
>
> For clarity I'll add the following paragraph:
>
>   The specification of an upgradeable Java SE module will indicate that
>   it is upgradeable.  If the specification of a Java SE module does not
>   indicate that the module is upgradeable then it is not upgradeable.
>
>>                                                 The standard can't
>> actually say anything about non-standard modules, but it could mandate
>> that every module is somehow tagged as upgradeable or not.
>
> It could, but that would require that we make upgradeability an explicit
> and standard property of all modules.
>
> Should we delay the release in order to design and implement that feature?

Is this an attempt to blackmail me :)

Of course I don't want to further delay the release. But I also don't
want that we simply reject useful features just because of a release
date. The fact that the feature of "upgradeability" also applies to
user modules which are linked into custom images (which you mentioned
above) makes this even more important.

The RI currently uses a non-standard feature (hashing) to enforce that
only specific modules can really be upgraded. Somebody who creates a
custom linked image that contains some library and application modules
needs to carefully think which of them should be upgradeable and which
should not. And he currently can not explicitly define that. If he
doesn't do anything, then his modules will be all upgradeable by
default (which may be not obvious). If he wants to make some of his
modules not upgradeable, he has to hash them when he creates his
custom image (which is also not obvious). And finally, he's not
allowed to hash the modules which should be upgradeable (which is
again not obvious). And we can't even advise that here in the
specification because hashing is not a standard feature.

So I think we should at least record these two issues ("making
upgradeability an explicit standard property" and "hashing of
dependent modules in order to ensure integrity") somewhere (e.g.
something like the JPMS issue summary) for future updates of this
specification.


And finally a new point which I only became aware of recently. What's
the relation of upgradeable and automatic module? I.e. can a plain old
jar file be used to upgrade a module? In the current RI it can and I
think that this is crucial because the whole mechanism is actually
designed for APIs which are usable on their own or that correspond to
standards maintained outside of the JCP. The only requirement for this
to work is that the jar file is either named in such a way that its
automatically created module name matches with that of the module or
that the jar file explicitly uses a corresponding
Automatic-Module-Name attribute in its manifest. So maybe you could
add a sentence like "a module can also be upgraded by a plain old jar
file if the automatic module name of the jar file is the same like the
name of the module"?

Regards,
Volker

>
> - Mark