Stability of lambda serialization
Brian Goetz
brian.goetz at oracle.com
Mon Aug 5 16:52:29 PDT 2013
I understand your objection. This was a difficult issue, and one which occupied a frustratingly large amount of time for the expert group. This is an area where there are no good solutions, just tradeoffs between different levels of user surprise, stability, and complexity. Different parties may have different preferences due to their own subjective ordering of the undesirability of different failure modes, and therefore may reach different conclusions about which solutions are more or less desirable. The consensus of the EG has been -- with the sole exception of Red Hat -- that the current target is the least bad of the options proposed, and that's what we chose to pursue.
Named lambdas were included on the list of issues gathered at the recent meeting to consider for the next round. Indeed, named lambdas would address this problem once and for all -- and we all knew this when it was raised the previous several times it came up. We left room to pursue them later, but for the time being, the EG felt that the syntactic overhead did not pull its weight.
The idea of restricting lambda serialization to non-capturing lambdas gained zero traction when it was proposed, in part because it makes things more confusing for the user, who now has to reason about which lambdas are capturing (its not like C++, where capturing lambdas are syntactically distinct from non-capturing ones).
It is unfortunate that Red Hat sees fit to threaten a No vote because an extensively discussed decision went in a direction they did not prefer. But the reality is, this issue *was* extensively discussed, Red Hat had many opportunities to attempt to win other EG members over to their preference, and the EG still -- after reopening the issue multiple times -- preferred the "weakly serializable" guarantees offered by the current model.
On Aug 5, 2013, at 3:15 PM, David M. Lloyd wrote:
> This was and is a bad decision. Some things serialize better than others. It makes far more sense to attempt to serialize as well as the things which serialize in the best possible way (i.e. fewest known issues). By simply requiring stable names for all serializable state, you achieve the best possible result.
>
> Justifying one bad solution because of another existing bad solution is bad engineering, and results in perpetuation of bad software. There's absolutely no reason to settle for this. There are simply no benefits to it other than the emotional comfort of being similar to something that exists, which is a foolish benchmark. If anonymous inner classes were not implemented in this manner, why would that make a difference to the way lambdas are serialized?
>
> The password example is deliberately chosen to be a simple expression of the potential problem. I imagine much more sophisticated problems resulting from modified sort order, information exposure from inadvertent capture, and so on. It's so much simpler to simply forbid serializing capturing lambdas, and much safer besides. On the other hand, I don't think we have really captured any compelling use cases for supporting serializable, capturing lambdas other than a lot of hand-wavy kinds of things.
>
> On 08/05/2013 02:08 PM, Remi Forax wrote:
>> You really want to serialize a password :) Funny idea.
>>
>> Anyway, we all know that serialization is broken, in many ways.
>> Usually, the only scenario that works is when the same code is used by
>> on both side of the wires,
>> trying to do something different is like playing the Russian roulette.
>> It may works for some time.
>>
>> If you change the code of an inner classes, you will have the same issue
>> as the one you describe
>> with lambdas. You may argue that lambdas should be better than inner
>> classes,
>> but given that even if we find a way to make the serialization stable to
>> this kind of change
>> there are still a lot of changes, like by example switching two lambdas
>> in the same method,
>> that makes the serialization not stable.
>>
>> So the EG had decided that serialization of lambdas should be not better
>> and not worst
>> than serialization of inner classes.
>>
>> Rémi
>>
>>
>> On 08/05/2013 08:09 PM, Scott Stark wrote:
>>> So Red Hat will go on the record to state that if serialization of
>>> lambdas is beyond the scope that DML suggested, we will vote no on the
>>> JSR. As a concluding remark on issue, consider an alternate form of
>>> the example given where the user and password variable types do not
>>> differ. In this situation, the reordering results in the password seen
>>> as the user. I can see that this will be the subject of yet another
>>> CVE somewhere down the road.
>>>
>>>
>>> Running testReadLambda after reorder...
>>> -> User secret1password <- -> Password bob <-
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: "Paul Benedict" <pbenedict at apache.org>
>>> To: lambda-spec-observers at openjdk.java.net
>>> Cc: "Scott Stark" <sstark at redhat.com>,
>>> lambda-spec-experts at openjdk.java.net
>>> Sent: Saturday, August 3, 2013 4:54:58 PM
>>> Subject: Re: Stability of lambda serialization
>>>
>>> I imagine this will mostly impact EE applications in a very negative way.
>>> It's one thing if my method signatures change, or I explicitly add some
>>> fields, but now adding/removing a local lambda will do that?
>>> Serialization
>>> errors are the worst thing in EE -- and this kind of just heaps on more
>>> misery.
>>>
>>> Paul
>>>
>>>
>>> On Sat, Aug 3, 2013 at 2:00 AM, Brian Goetz <brian.goetz at oracle.com>
>>> wrote:
>>>
>>>> This was discussed in the meeting after JVMLS this week. The consensus
>>>> was that, while *this particular* issue could be addressed, there is an
>>>> infinite spectrum of similar issues that cannot be addressed, and
>>>> that it
>>>> is preferable to draw a clean line about what the user can expect in
>>>> terms
>>>> of code changes destabilizing lambdas.
>>>>
>>>> That line is: If the code inside a method, or the method's signature,
>>>> changes *in any way*, lambdas captured in that method should be
>>>> considered
>>>> destabilized. Changes to other methods, or changes to the order of
>>>> methods, do not affect lambdas in an unchanged method.
>>>>
>>>> On Jul 23, 2013, at 10:35 AM, Scott Stark wrote:
>>>>
>>>>> Red Hat has a concern regarding how fragile the default serialization
>>>> behavior of lambda expressions is in the current reference
>>>> implementation,
>>>> currently:
>>>>> ironmaiden:OpenJDK starksm$
>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java
>>>> -version
>>>>> java version "1.8.0-ea"
>>>>> Java(TM) SE Runtime Environment (build 1.8.0-ea-b98)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.0-b40, mixed mode)
>>>>>
>>>>> The problem is that the serialized form of a lambda expression depends
>>>> on the order in which captured arguments are declared. The attached
>>>> simple
>>>> example demonstrates how easy it is for a trivial reordering of the
>>>> lambda
>>>> code block to result in an inability to deserialize a previously saved
>>>> expression.
>>>>> To produce this exception:
>>>>> 1. Run the serialization.AuthenticationContext.testWriteLambda method
>>>> with the lambda expression written as:
>>>>> Authenticator a = (Authenticator & Serializable) (String
>>>>> principal,
>>>> char[] pass) -> {
>>>>> // Run with p declared first when writing out the
>>>> /tmp/testWriteLambda.bin, then switch
>>>>> // to declare u first when running testReadLambda
>>>>> String p = "-> Password " + password + " <-";
>>>>> String u = "-> User " + user + " <-";
>>>>> return u + " " + p;
>>>>> };
>>>>> 2. Change the lambda expression to:
>>>>> Authenticator a = (Authenticator & Serializable) (String
>>>>> principal,
>>>> char[] pass) -> {
>>>>> // Run with p declared first when writing out the
>>>> /tmp/testWriteLambda.bin, then switch
>>>>> // to declare u first when running testReadLambda
>>>>> String u = "-> User " + user + " <-";
>>>>> String p = "-> Password " + password + " <-";
>>>>> return u + " " + p;
>>>>> };
>>>>>
>>>>> Recompile and run serialization.AuthenticationContext.testReadLambda to
>>>> produce:
>>>>> java.io.IOException: unexpected exception type
>>>>> at
>>>> java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538)
>>>>
>>>>> at
>>>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1110)
>>>>> at
>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807)
>>>>
>>>>> at
>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
>>>>> at
>>>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>>>>> at
>>>> serialization.AuthenticationContext.testReadLambda(AuthenticationContext.java:34)
>>>>
>>>>> ...
>>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>
>>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>>> at
>>>> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:222)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>
>>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>>> at
>>>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104)
>>>>> ... 30 more
>>>>> Caused by: java.lang.IllegalArgumentException: Invalid lambda
>>>> deserialization
>>>>> at
>>>> serialization.AuthenticationContext.$deserializeLambda$(AuthenticationContext.java:1)
>>>>
>>>>> ... 40 more
>>>>>
>>>>> One does not see the same level of sensitivity to the ordering of the
>>>> serialization fields in a POJO as demonstrated by the
>>>> serialization.AuthenticationContext.testWritePOJO/testReadPOJO cases
>>>> where
>>>> one can reorder the TestPOJO.{user,password} fields without having
>>>> serialization fail.
>>>>> We would like to see at least that level of stability of the serialized
>>>> form of lambda expressions.
>>>>> <AuthenticationContext.java><TestPOJO.java><Authenticator.java>
>>>>
>>>
>>
>
>
> --
> - DML
More information about the lambda-spec-observers
mailing list