Serializable lambda and bytecode rewriters

[Remi Forax]

Hi guys,
there is, i think, a serious issue in the way the deserialization of 
lambda is done by javac (or ecj which uses the same scheme), exactly in 
the code of the generated method $deserializeLambda$. Bytecode rewriting 
tools like jarjar or proguard works by rewriting all reference to types 
in class files so they need to distinguish if a reference is let say a 
String of a method descriptor. The actual code of $deserializeLambda$ 
generated by javac breaks that assumption, all parameters of a 
deserialized lambda are checked as String thus are not rewritten 
correctly by the bytecode rewriters.

So currently, it's no possible to use a bytecode rewriting tools with 
the jdk8 if there is a serializable lambda somewhere in the code.
My bad on that, I should have figure out that before the release of 8.

To fix this issue, the parameters that reference a lambda inside 
$deserializeLambda$ should be typed as constant method handles and 
constant method types but given that the serializable form a lambda is a 
bunch of strings, doing the mapping between the two forms is not that 
obvious.
Moreover, there is maybe a performance problem because
- the switch on the String that represent the name of the implementation 
method can not be used because it encodes the hashCode of a name that 
should be rewritten by the static tools.
- constant method handles and constant method types are not required to 
be interned by the VM thus may be re-created each time 
$deserializeLambda$ is called if the code use ldc.

The invokedynamic evangelist in me think that $deserializeLambda$ can be 
implemented as a unique invokedynamic call taking a SerializedLambda as 
parameter with all the parameters of all the serializable lambdas 
declared in the current class as bootstrap arguments. The bootstrap 
method will create a HashMap of a all methods names,
and at runtime check if a serialized lambda is valid or not.

Something like:
static class LambdaParameters {
   ...
   public boolean verify(SerializedLambda  serializedLambda) {
     // check all parameters
   }
}

public static CallSite deserializeLambda(Lookup lookup, String name, 
MethodType type, Object[] lambdaParameters) {
   HashMap<String, LambdaParameters> map = new HashMap<>();
   for(int i=0; i < lambdaParameters.length; i+=3) {
     ...
     MethodHandle mh = (MethodHandle) lambdaParameters[i + 1];
     String name = lookup.revealDirect(mh).getName();
     ...
     map.put(name, new LambdaParameters(...));
   }
   return new ConstantCallSite(CHECK.bindTo(map));
}
private static final MethodHandle = ::check;

private static boolean check(HashMap<String, LambdaParameters> map, 
SerializedLambda serializedLambda) {
   LambdaParameters parameters = 
map.get(serializedLambda.getImplMethodName());
   return parameters != null && parameters.verify(serializedLambda);
}

A bootstrap method can not have more than 252 arguments, so you can not 
encode more than 84 (252 / 3) serializable lambdas,
but I think that current scheme because it generated a lot of bytecodes 
will hit the 65 535 bytecodes near that number too.
Anyway, if there is more than 84 serializable lambdas, 
$deserializableLambda$ can call more than one invokedynamic in chain.

As you can see, the solution seems complex but may be somebody has a 
better solution :)

regards,
Rémi