Turning references into values

[Stephen Colebourne]

Java has always had a split between primitives and reference types.
This has always been awkward, often resulting in special cases to
handle the primitives. As such, I remain wary of adding value types,
which in essence expand the area of awkwardness within the language.
To combat this I propose a thought experiment, whereby reference types
become more like value types.

Lets consider that Valhalla adds a new value type - `Ref`. The generic
argument would be any reference type, with the implementation of Ref
ensuring the reference is non-null:

 Ref<ref T>

Lets also consider that Valhalla adds a new value type - `Opt` - with
a generic argument that can be any value type (this may be the
existing `Optional` class, or it may not). It would operate like
`Optional`, either containing a value, or being empty:

 Opt<val T>

Value types with a reference typevar are planned to be erased.
However, I propose that Ref<T> is different, with the typevar being
reified instead. This combination seems to yield some interesting
interactions with generics and erasusre:

 List<String> - an erased list of nullable Strings, as today

 List<Ref<String>> - a reified list of non-null Strings

 List<Opt<Ref<String>>> - a reified list of optional Strings

With these two tools, developers would have the ability to create
parameterized types reified with any type, not just reified with value
types. This is highly appealing, because it provides a clear and
obvious bridge between value and reference types under the developers
control. (And in many ways, a "reference" is a "value" - this change
just makes it explicit).

Unfortunately, the syntax involved is heavy, with long signatures and
ugly factory/access:

 List<Ref<String>> list = new ArrayList<>();
 list.add(Ref.of("alpha"));
 list.add(Ref.of("beta"));

 for (Ref<String> ref : list) {
   System.out.println(ref.value());
 }

But this fundamentally only a syntax issue. The following would be equivalent:

 List<String!> list = new ArrayList<>();
 list.add("alpha");
 list.add("beta");

 for (String! str : list) {
   System.out.println(str);
 }

The syntax `String!` would imply `Ref<String>`.
The syntax `String?` would imply `Opt<Ref<String>>`.
Boxing and unboxing would also be needed.
Declaring an `Opt` variable as null, would be equivalent to `Opt.empty`.
There would also need to be implicit conversion rules to allow
assignments from `Ref` to `Opt` to "classic-style", with some form of
cast for the opposite direction.
Note that developers using a String! or String? would not be able to
synchronize or use == etc as they would actually be working with
Ref<String>, as Ref is a value type.

With this in place, Java would have a mechanism to promote reference
types to act like value types, in a way that is opt-in. And when
opting-in, developers also get the ability to declare types as
nullable or non-null.

If there is a simple way to promote reference types to act like value
types (reified, no synchronization, no identity) then it effectively
provides a way to effectively _deprecate_ reference types (as
currently experienced). This might affect the choices made in altering
APIs like collections. It might also affect the choice as to whether
developers should be able to overload a method taking a parameterized
type (where overload by value type might be permitted, but not
overload by erased reference type).

This approach might also be used to "save" `Integer`, `Long` and
friends. While it is obviously possible to do reference operations on
`Integer`, it would not be possible to do them on `Integer!`. So,
perhaps `Integer!` would be a synonym for `int`.

In implementation terms, I believe that `Ref` and `Opt` would be
simple wrappers around the JVM-level reference, taking no more memory
at runtime. The only difference would appear to be that the type is
reified via a Ref wrapper object rather than being erased.

Where the thought experiment stops is consideration of the impact of
inheritance in reference types, and how that affects `Ref` and
parameterized types. I suspect there are issues lurking there.
Additionally, there is the key question of whether to aggressively
migrate the core-libs modules to use Ref and Opt everywhere (with
module linking used to smooth migration).

Of course, ultimately, the application may need to be able to get at
the reference-ness of the reference type instance. But this is surely
the rare edge case. Similarly, it seems that reflection-based
tools/frameworks/libraries would cope adequately with wrapper types,
just as they do with `Optional` today (and these tools will likely
need major work anyway for value types).

One difficulty with the plan is that there is still a mismatch between
value and reference syntax:

 RefType - a nullable reference type as Java SE 8
 RefType? - an optional ("nullable") value-style reference type
 RefType! - a non-null value-style reference type

 ValType - a non-null value type
 ValType? - an optional ("nullable") value type
 ValType! - disallowed, which is odd

There would be two ways to address this mismatch.

The first alternative is to say that all value type variables must be
declared with the ! suffix. Thus, a variable of the `Point` value type
would have to be referred to as `Point!`.  ie. it would be a compile
error to declare a variable of `Point`. This is not unreasonable, and
highlights the nullability of every type in the system.

The second alternative is to change the default in Java source, such
that `Foo` (reference or value) means non-null, `Foo?` means
optional/nullable, and `Foo!!%!!` means historic undefined/erased type
(insert ugly syntax here). This change could be done on a
class-by-class or module-by-module basis via a flag of some kind. I
think this would be the better approach for Java in the long run, at
the cost of pain in the short term (An IDE or other tool, could of
course rapdly change source code to add `?` to the end of each type in
a "migrate to Valhalla" operation.)

(It should be noted that I view the nullability mismatch of reference
and value types for a variable of type `Foo` to be a problem for the
current Valhalla proposals.)

In summary, I think this approach is worth considering because it
provides a way to bridge the schism betwen reference and value types.
Rather than treating them as two separate worlds with different rules,
it models references accurately, as a value pointing to an instance.
ie. references are simply another value type.

Stephen