PROPOSAL: Simplified Varargs Method Invocation

Thu Mar 5 21:39:06 PST 2009

Heh, that was the next proposal I was going to write.

(Apologies for a long and technically complicated post. I've rewritten  
it several times in an attempt to get to the point faster, but this is  
the best I can do).

It really pains me to rain on your parade. I was doing some  
experimenting and found an admittedly very unlikely scenario that  
isn't backwards compatible. Then I realized, that this is actually  
(arguably) a bug, or at least a severe misfeature in javac right now!  
Therefore, this proposal should fix this problem as well. Of all  
proposals so far, I rate this one the highest, because it causes so  
much confusion and is closer to a bug in javac than a language  
feature, so I would be very happy if this proposal can be fully ironed  
out.

Complete code showing the problem - copy and paste into Main.java,  
compile, and run:

------

import java.util.*;

class Foo<T> {
	public void foo(T... t) {}
}

class Bar extends Foo<String> {
         //de-varargsing? Why is this legal?
	public void foo(String[] t) {
		System.out.println(t.getClass().getComponentType());
	}
}

public class Main {
	public static void main(String[] args) {
		Foo<String> f = new Bar();
		List<String> l = Arrays.asList("a", "b", "c");
		bar(f, l);
	}

	public static <T> void bar(Foo<T> f, List<T> l) {
		f.foo(l.get(0), l.get(1), l.get(2));
	}
}
-----

The result is an error:

Exception in thread "main" java.lang.ClassCastException:  
[Ljava.lang.Object; cannot be cast to [Ljava.lang.String;

Right now, you get the usual varargs warning, but nothing else at  
compile time. At runtime, you get the above error.

The real bug here is the ability for any subclass to de-varargs a  
parameter. This bit has nothing to do with the proposal, other than  
that it so happens to deal with varargs.  De-varargsing a parameter is  
somewhat similar to a subclass tightening the type of an overriden  
parameter, which isn't legal / gives you a different method altogether  
instead of overriding.

  If you really want to pass varargs into such a method, then the only  
solution is to cast the object to a less specific type, and then all  
of a sudden it works. In other words, this can happen, right now, in  
java 1.5/1.6:

SubType s = new SubType();
s.foo(a, b, c); //method not found.
SuperType st = s;
st.foo(a, b, c); //no problem.

I say that's a bug.

The more serious problem, and one directly related to the proposal, is  
trying to reify a superclass's array type. (the subclass in the  
example above turning Foo's T into a reified 'String'). This action is  
fundamentally broken, and yet there's no way to avoid it. You can't  
effectively override a method of a superclass containing a generics  
type as an array's component type if you reified that generics  
parameter. You can try, but many callers simply can't call your method  
anymore. Right now, in java 1.5/1.6, you do get that generics warning  
Bob Lee's proposal is trying to move to the callee site, though it's  
not very informative and there's absolutely nothing a caller can do  
whatsoever. If Bob Lee's proposal is implemented, the warning just as  
useless, and arguably skipped even faster than it is now.

  Specifically, you can't:

A: widen the type of the parameter in the overriding method's  
signature to Object[] or Object..., as javac will refuse to compile it  
due to two different signatures erasing to the same signature.

B: add a bogus <T extends String> type and use that T. Javac will  
compile it, but the signature of that method is then foo(String[]),  
and thus its a completely separate method, that doesn't override  
parent's foo method.

Therefore, addendum #1 to this proposal:

Generate more warnings.

A. If you are overriding a method where in your signature an array is  
not varargsed, but in any of your supertypes' signatures, it is  
varargsed. This warning should be very severe as well and suggest in  
no uncertain terms that you NEED to replace your [] with "...";  
optimally this is a compile time error, but fixing this now wouldn't  
be migration compatible. This warning should not show up if using - 
source 1.4 or lower, for obvious reasons.

B. If you are overriding a method where in your signature an array  
type's component type has been reified compared to any of your  
supertypes' signatures for this method. The warning should explain  
that there's no work-around, but that anybody treating your instance  
as a type of (X) will likely get a runtime error if they tried to call  
this method. The only fix I can think of is to rejigger the signatures  
somehow: The method should signature as an Object[] same as parent,  
but javac should NOT consider this a separate method that erases to  
the same signature - it IS intended as the same method. If this is  
done, the warning on the method should be translated to something  
along the lines of Bob Lee's proposed warning (the actual component  
type of the array, if asked for at reification, isn't neccessarily  
correct, and your string array might all of a sudden give you integers  
if the caller handed you a generics-tainted array).

A second addendum I would strongly urge is to use escape detection to  
eliminate the warning on the var-args-using methods. The following  
operations are all entirely safe, and they are also the only ones used  
by every single varargs method I've ever laid my eyes on, except  
Arrays.asList:

Safe #1: Ask for the vararg array's length. Obviously safe - length  
has no relation to the array's type.

Safe #2: Member lookup (array[index]). The type of a member lookup is  
entirely dependent on the compile-time type. The caller needs to worry  
about that compile-time type too, so can't stuff mistyped things in  
that varargs array without triggering a generics warning of their own.  
Once somebody somewhere gets a generics warning, any resulting  
ClassCastExceptions are acceptable (that's the status quo now, after  
all). Non-generified lookups such as the example above can't even run,  
because the method invocation itself will already blow up. This is  
bad, but it happens now too.

Safe #3: foreaching through a varargs parameter. foreaching a  
combination of .length and member lookups, so by induction is  
obviously as safe as #1 and #2.

Anything else, -anything- at all, including using it in any expression  
(other than the safe forms above), storing it into another variable or  
field, using it as a parameter to another method, or calling any other  
property on it, including toString() (there's no point in calling  
toString() on an array anyway, so why bother declaring that safe?) -  
is unsafe, and would generate the warning. The warning is then not  
generated on the var-args-accepting-method's signature, but instead on  
the first node in that method's AST that does something that isn't  
safe to the varargs parameter.

The only borderline cases where the above preconditions are broken is  
Arrays.asList, which is a unique exception, and varargs methods that  
pass the entire varargs parameter to a helper method. It might be  
possible to declare passing varargs parameters to another method that  
also takes varargs as safe, but I'm not 100% certain that's always  
safe (as safe as generics are going to be in a non-reified world, that  
is).

Note that for most of these cases, there are mixed-version problems.  
New code that calls into old code means nobody gets any warnings. I'm  
not sure if Bob's proposal includes this, but the compiler should  
continue to generate warnings on the site of the caller if the class  
file format of the targeted method goes with javac v1.6 or below.

If a code-base starts mixing generated class files, things get even  
worse; methods that, during compile-time, weren't overriding anything,  
are suddenly supposed to override a generics or varargs based  
superclass because just the superclass has been recompiled with  
changes, for example. The only argument I can give in favour of this  
proposal is that doing that sort of thing already breaks a number of  
apparent invariants and can give you NoSuchMethodErrors and the like.

  --Reinier Zwitserloot

On Mar 6, 2009, at 02:30, Bob Lee wrote:

> Simplified Varargs Method Invocation
> AUTHOR: Bob Lee
>
> *OVERVIEW
> *
> FEATURE SUMMARY: When a programmer tries to invoke a *varargs*  
> (variable
> arity) method with a non-reifiable varargs type, the compiler  
> currently
> generates an "unsafe operation" warning. This proposal moves the  
> warning
> from the call site to the method declaration.
>
> MAJOR ADVANTAGE: Safely and significantly reduces the total number of
> warnings reported to and suppressed by programmers. Reduces programmer
> confusion. Enables API designers to use varargs.
>
> MAJOR BENEFIT: Plugs a leaky abstraction. Creating an array when you  
> call a
> varargs method is an implementation detail that we needn't expose to  
> users.
> Addresses a well known "gotcha" encountered when you mix varargs and
> generics. Most programmers are surprised to find out you can only  
> clear this
> warning by suppressing it (especially considering we introduced  
> varargs and
> generics in the same language update); they waste time looking for
> alternatives that don't exist. Google Code Search finds almost 90k  
> callers
> of Arrays.asList() (http://tinyurl.com/dept4d). We could safely  
> suppress the
> warning once and for all on asList()'s declaration instead of  
> unnecessarily
> warning every caller that uses a non-reifiable type.
>
> MAJOR DISADVANTAGE: The compiler will generate a warning for a method
> declaration whether or not someone actually calls the method with a
> non-reifiable type. Allows loss of type safety if the varargs method
> suppresses the warning and uses the varargs array unsafely.
>
> ALTERNATIVES:
>  a) Don't mix varargs with generics. Use the more verbose and less
> straightforward but warning-free builder pattern. Most API designers  
> choose
> this route.
>  b) Improve the warning message. Current message: "uses unchecked or  
> unsafe
> operations"
>  c) Reify generics.
>  d) Introduce a second varargs syntax (perhaps using "...." instead of
> "...") that uses List<T> instead of T[].
>  e) Defile the type system.
>
> Note: This proposal doesn't preclude any of these other approaches.
>
> *EXAMPLES
> *
> Before this change:
>
>  static <T> List<T> asList(T... elements) { ... }
>
>  static List<Callable<String>> stringFactories() {
>    Callable<String> a, b, c;
>    ...
>    *// Warning: **"uses unchecked or unsafe operations"*
>    return asList(a, b, c);
>  }
>
> After this change:
>
>  *// Warning: **"enables unsafe generic array creation"*
>  static <T> List<T> asList(T... elements) { ... }
>
>  static List<Callable<String>> stringFactories() {
>    Callable<String> a, b, c;
>    ...
>    return asList(a, b, c);
>  }
>
> If asList() prohibits storing elements that aren't of type T in the  
> elements
> array, we can safely suppress the warning:
>
>  *@SuppressWarnings("generic-varargs")
>  // Ensures only values of type T can be stored in elements.
> *  static <T> List<T> asList(T... elements) { ... }
>
> *DETAILS
> *
> SPECIFICATION: When compiling code that calls a varargs method with a
> non-reifiable varargs element type, if the target method was compiled
> targeting Java 7 or later, the compiler needn't generate a warning  
> for the
> caller. When compiling a varargs method that could accept a non- 
> reifiable
> varargs type, the compiler should generate a warning on the varargs  
> method
> declaration. A varargs type is non-reifiable if it contains a type  
> variable
> anywhere in its signature. For a varargs argument of type T, the  
> programmer
> can safely suppress the warning using @SuppressWarnings("generic- 
> varargs")
> so long as the varargs method ensures that only elements of type T  
> can be
> stored in the varargs array.
>
> COMPILATION: Tools should no longer generate a warning for varargs  
> method
> callers. Instead, they should generate a warning for varargs method
> declarations that support non-reifiable types.
>
> TESTING: Compile test programs and ensure that the compiler  
> generates the
> expected warnings.
>
> LIBRARY SUPPORT: Suppress warnings on the following varargs methods  
> in the
> JDK:
>  - Arrays.asList(T... a)
>  - Collections.addAll(Collection<? super T> c, T... elements)
>  - EnumSet.of(E first, E... rest)
>
> REFLECTIVE APIS: n/a
>
> OTHER CHANGES: n/a
>
> MIGRATION: Existing callers may be able to remove
> @SuppressWarnings("unchecked") from their code. Existing libraries  
> should
> add @SuppressWarnings("generic-varargs") to methods with signatures
> containing non-reifiable varargs types.
>
> *COMPATIBILITY
> *
> BREAKING CHANGES: None
>
> EXISTING PROGRAMS: If you recompile an existing program with "- 
> target 7",
> the compiler will generate warnings for method declarations containing
> non-reifiable varargs types.
>
> *REFERENCES
> *
> EXISTING BUGS: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6227971
>
> JLS Section 15.12.4.2 "Evaluate Arguments" (
> http://java.sun.com/docs/books/jls/third_edition/html/ 
> expressions.html)
>
> "If the method being invoked is a variable arity method
> (§8.4.1)<http://java.sun.com/docs/books/jls/third_edition/html/classes.html#38698 
> >m,
> it necessarily has n>0 formal parameters. The final formal parameter
> of m
> necessarily has type T[] for some T, and m is necessarily being  
> invoked with
> k0 actual argument expressions.
>
> If m is being invoked with kn actual argument expressions, or, if m  
> is being
> invoked with k=n actual argument expressions and the type of the kth
> argument expression is not assignment compatible with T[], then the  
> argument
> list (e1, ... , en-1, en, ...ek) is evaluated as if it were written  
> as (e1,
> ..., en-1, new T[]{en, ..., ek}).
>
> The argument expressions (possibly rewritten as described above) are  
> now
> evaluated to yield argument values. Each argument value corresponds to
> exactly one of the method's n formal parameters."
>
> Angelika Langer's Java Generics FAQ, "Why does the compiler  
> sometimes issue
> an unchecked warning when I invoke a 'varargs' method?" (
> http://tinyurl.com/8w2dk)
>
> Josh Bloch's "Effective Java" 2nd Edition, page 120 (
> http://tinyurl.com/chtgbd)
>
> Alex Miller's blog, "Generics puzzler - array construction" (
> http://tech.puredanger.com/2007/02/27/generics-array-construction/)
>
> "Java Generic and Collections", page 95 (http://tinyurl.com/c53pnu)
>