Stability of lambda serialization

[Remi Forax]

On 07/27/2013 07:18 PM, Brian Goetz wrote:
> Yes, when this came up the first several times, the consensus of the 
> EG was that we would build on the de-facto, practical constraint that 
> people have learned to live with for years when dealing with 
> serialization in the presence of inner classes: make sure the same 
> classfile bits are on both sides of the pipe.
>
> Inner classes have a naming instability, where you have a stable 
> method name but an unstable class name.  Lambdas have the opposite, 
> but effectively equivalent instability: a stable class name but an 
> unstable method name.  People have learned to use serialization with 
> inner classes by the technique of keeping the classfile bits the same 
> on both sides of the wire.  The exact same technique enables 
> serialization of lambdas.
>
> In particular, the EG rejected the extremes of "make it perfect" and 
> "don't allow it at all" as being "the perfect is the enemy of the 
> good."  Inner classes have been exactly this hostile to serialization 
> for 15 years, but people who want to use serialization have learned 
> coping techniques to do so.  The compromises in the middle are squishy 
> but better than either extreme.
>
> Of course, if we can make it better without exposing additional 
> complexity, so much the better.  For example, we did a few rounds of 
> this by encoding the hash of the method name and signature so that 
> simply reordering the methods in a class did not trigger such 
> instabilities.  But the goal was never perfect stability; it was just 
> "at least as good as inner classes."  I believe we've reached that goal.
>
> Do you have a concrete proposal for how we would improve lambda 
> serialization to be robust in the face of changes in order of capture? 
> On the surface it looks entirely possible.

First, I hope there is not a hashtable somewhere in javac and that the 
order of capture is really defined by the order of the captured variables
inside the body of the lambda. Also, I find this order better than the 
order of the declared captured variable in the outer scopes.

The only solution I see for that is to also encodes the name of the 
captured variables (we do something like this for enums)
and to encode the name of the parameter of the generated method lambda$$$$.
It will make the serialized binary form bigger and  the deserialization 
(a little, we can leverage invokedynamic here to do the shuffling) slower.
And anyway, if one local variable is renamed it will not worked.

I don't know if we want to go that far ...

cheers,
Rémi

>
> On 7/27/2013 12:52 PM, Sam Pullara wrote:
>> I think the requirement should be that you have the same classes on each
>> side of the wire.
>>
>> Sam
>>
>>
>> On Tue, Jul 23, 2013 at 10:35 AM, Scott Stark <sstark at redhat.com
>> <mailto:sstark at redhat.com>> wrote:
>>
>>     Red Hat has a concern regarding how fragile the default
>>     serialization behavior of lambda expressions is in the current
>>     reference implementation, currently:
>>     ironmaiden:OpenJDK starksm$
>> /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java 
>> -version
>>     java version "1.8.0-ea"
>>     Java(TM) SE Runtime Environment (build 1.8.0-ea-b98)
>>     Java HotSpot(TM) 64-Bit Server VM (build 25.0-b40, mixed mode)
>>
>>     The problem is that the serialized form of a lambda expression
>>     depends on the order in which captured arguments are declared. The
>>     attached simple example demonstrates how easy it is for a trivial
>>     reordering of the lambda code block to result in an inability to
>>     deserialize a previously saved expression.
>>
>>     To produce this exception:
>>     1. Run the serialization.AuthenticationContext.testWriteLambda
>>     method with the lambda expression written as:
>>            Authenticator a = (Authenticator & Serializable) (String
>>     principal, char[] pass) -> {
>>                // Run with p declared first when writing out the
>>     /tmp/testWriteLambda.bin, then switch
>>                // to declare u first when running testReadLambda
>>                String p = "-> Password " + password + " <-";
>>                String u = "-> User " + user + " <-";
>>                return u + " " + p;
>>            };
>>     2. Change the lambda expression to:
>>            Authenticator a = (Authenticator & Serializable) (String
>>     principal, char[] pass) -> {
>>                // Run with p declared first when writing out the
>>     /tmp/testWriteLambda.bin, then switch
>>                // to declare u first when running testReadLambda
>>                String u = "-> User " + user + " <-";
>>                String p = "-> Password " + password + " <-";
>>                return u + " " + p;
>>            };
>>
>>     Recompile and run serialization.AuthenticationContext.testReadLambda
>>     to produce:
>>
>>     java.io.IOException: unexpected exception type
>>              at
>> java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538)
>>              at
>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1110)
>>              at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807)
>>              at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
>>              at
>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>>              at
>> serialization.AuthenticationContext.testReadLambda(AuthenticationContext.java:34)
>>     ...
>>     Caused by: java.lang.reflect.InvocationTargetException
>>              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
>> Method)
>>              at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>              at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>              at
>> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:222)
>>              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
>> Method)
>>              at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>              at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>              at
>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104)
>>              ... 30 more
>>     Caused by: java.lang.IllegalArgumentException: Invalid lambda
>>     deserialization
>>              at
>> serialization.AuthenticationContext.$deserializeLambda$(AuthenticationContext.java:1)
>>              ... 40 more
>>
>>     One does not see the same level of sensitivity to the ordering of
>>     the serialization fields in a POJO as demonstrated by the
>> serialization.AuthenticationContext.testWritePOJO/testReadPOJO cases
>>     where one can reorder the TestPOJO.{user,password} fields without
>>     having serialization fail.
>>
>>     We would like to see at least that level of stability of the
>>     serialized form of lambda expressions.
>>
>>