CONSTANT_Class_info and L-envelopes

[Frederic Parain]

Greetings,

After the discussions of the past weeks about checkcast and the use of
envelopes in CONSTANT_Class_info entries, a consensus has been reached
for the case of Q-types. However, the case of L-types (should they use
an L-envelope or not) is still subject to discussion. In one hand,
forcing L-type to use envelopes would be a nice unifying move, in the
other hand, it would have a bigger impact on tools, requiring a flag
day approach instead of a smooth transition.

The current approach is to continue not to use L-envelopes in
CONSTANT_Class_info entries, but because some people might want to
experiment with L-envelopes, it has been suggested to add a VM flag
to support them.

Having a global flag to require that all class files use the same
format (either L-envelopes everywhere or L-envelopes nowhere) seems
impractical because it would require to recompile all classes to
switch from one mode to the other.

Being more lenient and supporting both modes causes a new issue:
the double syntax would allow a symbolic reference to a class to
exist in two forms inside a single class file: one form with the
L-envelope, one form without the L-envelope, creating two distinct
constant pool entries. The problem with having two entries representing
a symbolic reference to the same class is that there’s no guarantee
that the resolution of these two entries will produce the same result.

One solution to prevent this to happen would be to verify that a given
class is never referenced through two different entries, but this would
add a lot of checks during class file parsing.

Another solution would be to have a “mode” per class file, either
always using L-envelopes, or never using them for CONSTANT_Class_info.
The class file doesn’t have to declare which mode it uses, this will
automatically detected by the JVM while processing the first
CONSTANT_Class_info entry, and then enforced for the remaining of the
processing of the class file. A global VM flag controls if the VM
must support the “L-envelope required” mode or reject it (default
is to reject it).

This solution has been prototyped here:
http://cr.openjdk.java.net/~fparain/new_constant_class/webrev.00/index.html

It has been tested with manually modified class files, but has no unit
tests provided with the changeset.

Before submitting a pull-request, I’d like to have some feedback about this
approach. If you think this approach makes sense, I can push it, otherwise
I’ll drop it.

Thank you,

Fred