From forax at univ-mlv.fr Sat Jul 6 02:20:36 2013 From: forax at univ-mlv.fr (Remi Forax) Date: Sat, 06 Jul 2013 11:20:36 +0200 Subject: Mixing declaration site variance and wildcards Message-ID: <51D7E164.8060403@univ-mlv.fr> Something that is worth considering if we want functional interfaces to be declared covariant/contravariant/invariant in Java.next http://www.cs.cornell.edu/~ross/publications/mixedsite/ R?mi From stephan.herrmann at berlin.de Sun Jul 7 05:30:03 2013 From: stephan.herrmann at berlin.de (Stephan Herrmann) Date: Sun, 07 Jul 2013 14:30:03 +0200 Subject: abstract method in indirect superclass Message-ID: <51D95F4B.2070708@berlin.de> Hi, I'm currently investigating an issue of required overriding of an abstract method, illustrated by this example: class A { } class B extends A { } abstract class X1 { abstract void foo(T1 t); } abstract class X2 extends X1 { @Override void foo(A t) { } } class X3 extends X2 { } The question is whether or not X3 must provide an implementation of "void foo(T3)", i.e., should an error be flagged against X3, because it does not inherit nor implement a method whose signature is a subsignature of "(T3)"? The way I read the 0.6.2 spec, the answer is "no", because X3 can only inherit methods from its direct superclass X2, but not from X1. Since X2 overrides foo, the abstract version is not relevant for X3. OTOH, in my intuitive understanding of JLS7 the following sentence would apply: "Any of C's superclasses has an abstract method and C neither declares nor inherits a method that implements (?8.4.8.1) it." Interpreting this sentence *in the context of C*, I see a method foo(T3) with no legal implementation. Finally, 0.6.2 sect. 8.1.1.1 says: "Discussion and motivation: These changes are not meant to substantively change the definition of an abstract class, but rather to clarify the existing meaning." By this comment, are you saying that the new interpretation has always been intended and the above example should be accepted for all versions of Java? Maybe this specific situation deserves a mentioning in this paragraph of discussion? Technically, the subsignature rule sneaks in a flavor of contravariant overriding, which probably wasn't explicitly intended, but at least it should cause no problems wrt soundness. thanks, Stephan From forax at univ-mlv.fr Sun Jul 7 07:44:08 2013 From: forax at univ-mlv.fr (Remi Forax) Date: Sun, 07 Jul 2013 16:44:08 +0200 Subject: abstract method in indirect superclass In-Reply-To: <51D95F4B.2070708@berlin.de> References: <51D95F4B.2070708@berlin.de> Message-ID: <51D97EB8.5040109@univ-mlv.fr> On 07/07/2013 02:30 PM, Stephan Herrmann wrote: > Hi, > > I'm currently investigating an issue of required overriding of > an abstract method, illustrated by this example: > > class A { } > class B extends A { } > > abstract class X1 { > abstract void foo(T1 t); > } > abstract class X2 extends X1 { > @Override void foo(A t) { } > } I don't see how this can compile despite the fact that both eclipse and javac are happy with this code :( foo(A) does not override foo(T2) if by example T2 is B so it should result in a name clash. cheers, R?mi > class X3 extends X2 { } > > The question is whether or not X3 must provide an implementation > of "void foo(T3)", i.e., should an error be flagged against X3, > because it does not inherit nor implement a method whose signature > is a subsignature of "(T3)"? > > The way I read the 0.6.2 spec, the answer is "no", because X3 > can only inherit methods from its direct superclass X2, but not > from X1. Since X2 overrides foo, the abstract version is not > relevant for X3. > > OTOH, in my intuitive understanding of JLS7 the following sentence > would apply: > > "Any of C's superclasses has an abstract method and C neither declares > nor inherits a method that implements (?8.4.8.1) it." > > Interpreting this sentence *in the context of C*, I see a method > foo(T3) with no legal implementation. > > Finally, 0.6.2 sect. 8.1.1.1 says: > "Discussion and motivation: > These changes are not meant to substantively change the definition > of an abstract class, but rather to clarify the existing meaning." > > > By this comment, are you saying that the new interpretation has > always been intended and the above example should be accepted for all > versions of Java? Maybe this specific situation deserves a mentioning > in this paragraph of discussion? > > > Technically, the subsignature rule sneaks in a flavor of > contravariant overriding, which probably wasn't explicitly intended, > but at least it should cause no problems wrt soundness. > > thanks, > Stephan From stephan.herrmann at berlin.de Sun Jul 7 08:03:40 2013 From: stephan.herrmann at berlin.de (Stephan Herrmann) Date: Sun, 07 Jul 2013 17:03:40 +0200 Subject: abstract method in indirect superclass In-Reply-To: <51D97EB8.5040109@univ-mlv.fr> References: <51D95F4B.2070708@berlin.de> <51D97EB8.5040109@univ-mlv.fr> Message-ID: <51D9834C.6040702@berlin.de> On 07/07/2013 04:44 PM, Remi Forax wrote: > I don't see how this can compile despite the fact that both eclipse and javac > are happy with this code :( In fact this surfaced because a recent clean-up in ecj incidentally changed this behavior so in 4.3 we no longer accept such code. > foo(A) does not override foo(T2) if by example T2 is B so it should result in a name clash. I don't think this part is at stake. The signature of foo(A) is a subsignature of the signature of foo(T2) because the former equals the erasure of the latter. Only what's going on in X3 is unclear in JLS7. My questions are whether the change made for JSR 335 has been made consciously and whether any changes should be made to compilers for Java 7-. thanks, Stephan From forax at univ-mlv.fr Sun Jul 7 08:36:46 2013 From: forax at univ-mlv.fr (Remi Forax) Date: Sun, 07 Jul 2013 17:36:46 +0200 Subject: abstract method in indirect superclass In-Reply-To: <51D9834C.6040702@berlin.de> References: <51D95F4B.2070708@berlin.de> <51D97EB8.5040109@univ-mlv.fr> <51D9834C.6040702@berlin.de> Message-ID: <51D98B0E.9030606@univ-mlv.fr> On 07/07/2013 05:03 PM, Stephan Herrmann wrote: > On 07/07/2013 04:44 PM, Remi Forax wrote: > > I don't see how this can compile despite the fact that both eclipse > and javac > > are happy with this code :( > > In fact this surfaced because a recent clean-up in ecj incidentally > changed this behavior so in 4.3 we no longer accept such code. > > > foo(A) does not override foo(T2) if by example T2 is B so it should > result in a name clash. > > I don't think this part is at stake. > The signature of foo(A) is a subsignature of the signature of foo(T2) > because the former equals the erasure of the latter. it should because this is obviously a bug, this code throws a CCE class A { } class B extends A { } class C extends A { } abstract class X1 { void foo(T1 t) { System.out.println("X1"); } } class X2 extends X1 { @Override void foo(A t) { System.out.println("X2"); } } class X3 extends X2 { @Override void foo(B t) { System.out.println("X3"); } } public class Test { public static void main(String[] args) { X2 x2 = new X3(); x2.foo(new C()); } } R?mi > > Only what's going on in X3 is unclear in JLS7. My questions are > whether the change made for JSR 335 has been made consciously and > whether any changes should be made to compilers for Java 7-. > > thanks, > Stephan > > From stephan.herrmann at berlin.de Sun Jul 7 09:46:18 2013 From: stephan.herrmann at berlin.de (Stephan Herrmann) Date: Sun, 07 Jul 2013 18:46:18 +0200 Subject: abstract method in indirect superclass In-Reply-To: <51D98B0E.9030606@univ-mlv.fr> References: <51D95F4B.2070708@berlin.de> <51D97EB8.5040109@univ-mlv.fr> <51D9834C.6040702@berlin.de> <51D98B0E.9030606@univ-mlv.fr> Message-ID: <51D99B5A.8030802@berlin.de> On 07/07/2013 05:36 PM, Remi Forax wrote: > On 07/07/2013 05:03 PM, Stephan Herrmann wrote: >> On 07/07/2013 04:44 PM, Remi Forax wrote: >> > I don't see how this can compile despite the fact that both eclipse and javac >> > are happy with this code :( >> >> In fact this surfaced because a recent clean-up in ecj incidentally >> changed this behavior so in 4.3 we no longer accept such code. >> >> > foo(A) does not override foo(T2) if by example T2 is B so it should result in a name clash. >> >> I don't think this part is at stake. >> The signature of foo(A) is a subsignature of the signature of foo(T2) >> because the former equals the erasure of the latter. > > it should because this is obviously a bug, > this code throws a CCE > > class A { } > class B extends A { } > class C extends A { } > > abstract class X1 { > void foo(T1 t) { > System.out.println("X1"); > } > } > class X2 extends X1 { > @Override void foo(A t) { > System.out.println("X2"); > } > } > class X3 extends X2 { > @Override void foo(B t) { > System.out.println("X3"); > } > } > > public class Test { > public static void main(String[] args) { > X2 x2 = new X3(); > x2.foo(new C()); > } > } Thanks for the interesting example. Here javac reports: Test.java:17: error: name clash: foo(B) in X3 overrides a method whose erasure is the same as another method, yet neither overrides the other @Override void foo(B t) { ^ first method: foo(A) in X2 second method: foo(T1) in X1 where T1 is a type-variable: T1 extends A declared in class X1 1 error For Java 7 this seems to be the right answer. The fact that ecj is silent appears to be a bug. I only wonder how this spells out in Java 8: The relevant condition in JLS 7, 8.4.8.3 reads: "The signature of m1 or some method m1 overrides (directly or indirectly) has the same erasure as the signature of m2 or some method m2 overrides (directly or indirectly)." Assuming that 8.4.8.3 itself is not changed for JLS8, we only have to make sure that "overrides" still includes the situation of a method in an indirect superclass that was already overridden in an intermediate class, viz., does X3.foo actually override X1.foo although X1.foo was already overridden by X2.foo? Looking at 8.4.8.1 from spec 0.6.2, I'd say yes, the error reported by javac is still valid. cheers, Stephan From daniel.smith at oracle.com Fri Jul 12 11:59:17 2013 From: daniel.smith at oracle.com (Dan Smith) Date: Fri, 12 Jul 2013 12:59:17 -0600 Subject: abstract method in indirect superclass In-Reply-To: <51D95F4B.2070708@berlin.de> References: <51D95F4B.2070708@berlin.de> Message-ID: Hi. These sort of questions are very useful, so keep them coming. I've been on vacation, so am just now taking a look at this. I immediately thought of this bug: http://bugs.sun.com/view_bug.do?bug_id=8010681 > OTOH, in my intuitive understanding of JLS7 the following sentence > would apply: > > "Any of C's superclasses has an abstract method and C neither declares > nor inherits a method that implements (?8.4.8.1) it." > > Interpreting this sentence *in the context of C*, I see a method > foo(T3) with no legal implementation. Our problem here is the definition of "implements". Your interpretation is that it should be tested by comparing signatures in the context of C. JLS 7 is vague on this (it is also vague on just how we define a "method"), but I think the most stable interpretation is that, generally, we're talking about the declaration-site signature of the method. In the special case of an inherited method that is "said to override" some additional methods, per 8.4.8.4, we're talking about the signature at the point that this overriding occurs. (In other words, the overriding relation is monotonic with respect to inheritance: you can add new overrides via inheritance, but can't "undo" existing ones.) The "In the context of C" interpretation doesn't work at all if we're talking about methods that are package-access, and thus not accessible to C. The behavior of javac 7 and ecj 3.8 supports this latter interpretation. That's not to say that we must definitively match what they do, but it does suggest that this approach is what was intended. In the 0.6.2 spec, we clarify the error condition by (mostly) making the check a test for abstract methods that are members of the class, with a special rule for package-access methods. This transformation relies on an assumption that "Any of C's superclasses has an abstract method and C neither declare nor inherits a method that implements (?8.4.8.1) it" iff either 1) the abstract method is a member of C; or 2) the abstract method is package-access and not overridden in C or any of its superclasses. > By this comment, are you saying that the new interpretation has > always been intended and the above example should be accepted for all > versions of Java? Maybe this specific situation deserves a mentioning > in this paragraph of discussion? Yes. I can add a discussion point. ?Dan From sstark at redhat.com Tue Jul 23 10:35:26 2013 From: sstark at redhat.com (Scott Stark) Date: Tue, 23 Jul 2013 13:35:26 -0400 (EDT) Subject: Stability of lambda serialization In-Reply-To: <286109594.871872.1374598513380.JavaMail.root@redhat.com> Message-ID: <572125674.885842.1374600926640.JavaMail.root@redhat.com> Red Hat has a concern regarding how fragile the default serialization behavior of lambda expressions is in the current reference implementation, currently: ironmaiden:OpenJDK starksm$ /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java -version java version "1.8.0-ea" Java(TM) SE Runtime Environment (build 1.8.0-ea-b98) Java HotSpot(TM) 64-Bit Server VM (build 25.0-b40, mixed mode) The problem is that the serialized form of a lambda expression depends on the order in which captured arguments are declared. The attached simple example demonstrates how easy it is for a trivial reordering of the lambda code block to result in an inability to deserialize a previously saved expression. To produce this exception: 1. Run the serialization.AuthenticationContext.testWriteLambda method with the lambda expression written as: Authenticator a = (Authenticator & Serializable) (String principal, char[] pass) -> { // Run with p declared first when writing out the /tmp/testWriteLambda.bin, then switch // to declare u first when running testReadLambda String p = "-> Password " + password + " <-"; String u = "-> User " + user + " <-"; return u + " " + p; }; 2. Change the lambda expression to: Authenticator a = (Authenticator & Serializable) (String principal, char[] pass) -> { // Run with p declared first when writing out the /tmp/testWriteLambda.bin, then switch // to declare u first when running testReadLambda String u = "-> User " + user + " <-"; String p = "-> Password " + password + " <-"; return u + " " + p; }; Recompile and run serialization.AuthenticationContext.testReadLambda to produce: java.io.IOException: unexpected exception type at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538) at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1110) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at serialization.AuthenticationContext.testReadLambda(AuthenticationContext.java:34) ... Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:222) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104) ... 30 more Caused by: java.lang.IllegalArgumentException: Invalid lambda deserialization at serialization.AuthenticationContext.$deserializeLambda$(AuthenticationContext.java:1) ... 40 more One does not see the same level of sensitivity to the ordering of the serialization fields in a POJO as demonstrated by the serialization.AuthenticationContext.testWritePOJO/testReadPOJO cases where one can reorder the TestPOJO.{user,password} fields without having serialization fail. We would like to see at least that level of stability of the serialized form of lambda expressions. From sam at sampullara.com Sat Jul 27 09:52:25 2013 From: sam at sampullara.com (Sam Pullara) Date: Sat, 27 Jul 2013 09:52:25 -0700 Subject: Stability of lambda serialization In-Reply-To: <572125674.885842.1374600926640.JavaMail.root@redhat.com> References: <286109594.871872.1374598513380.JavaMail.root@redhat.com> <572125674.885842.1374600926640.JavaMail.root@redhat.com> Message-ID: I think the requirement should be that you have the same classes on each side of the wire. Sam On Tue, Jul 23, 2013 at 10:35 AM, Scott Stark wrote: > Red Hat has a concern regarding how fragile the default serialization > behavior of lambda expressions is in the current reference implementation, > currently: > ironmaiden:OpenJDK starksm$ > /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java > -version > java version "1.8.0-ea" > Java(TM) SE Runtime Environment (build 1.8.0-ea-b98) > Java HotSpot(TM) 64-Bit Server VM (build 25.0-b40, mixed mode) > > The problem is that the serialized form of a lambda expression depends on > the order in which captured arguments are declared. The attached simple > example demonstrates how easy it is for a trivial reordering of the lambda > code block to result in an inability to deserialize a previously saved > expression. > > To produce this exception: > 1. Run the serialization.AuthenticationContext.testWriteLambda method with > the lambda expression written as: > Authenticator a = (Authenticator & Serializable) (String principal, > char[] pass) -> { > // Run with p declared first when writing out the > /tmp/testWriteLambda.bin, then switch > // to declare u first when running testReadLambda > String p = "-> Password " + password + " <-"; > String u = "-> User " + user + " <-"; > return u + " " + p; > }; > 2. Change the lambda expression to: > Authenticator a = (Authenticator & Serializable) (String principal, > char[] pass) -> { > // Run with p declared first when writing out the > /tmp/testWriteLambda.bin, then switch > // to declare u first when running testReadLambda > String u = "-> User " + user + " <-"; > String p = "-> Password " + password + " <-"; > return u + " " + p; > }; > > Recompile and run serialization.AuthenticationContext.testReadLambda to > produce: > > java.io.IOException: unexpected exception type > at > java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538) > at > java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1110) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > serialization.AuthenticationContext.testReadLambda(AuthenticationContext.java:34) > ... > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at > java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:222) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at > java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104) > ... 30 more > Caused by: java.lang.IllegalArgumentException: Invalid lambda > deserialization > at > serialization.AuthenticationContext.$deserializeLambda$(AuthenticationContext.java:1) > ... 40 more > > One does not see the same level of sensitivity to the ordering of the > serialization fields in a POJO as demonstrated by the > serialization.AuthenticationContext.testWritePOJO/testReadPOJO cases where > one can reorder the TestPOJO.{user,password} fields without having > serialization fail. > > We would like to see at least that level of stability of the serialized > form of lambda expressions. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-spec-experts/attachments/20130727/68ec1c64/attachment.html From brian.goetz at oracle.com Sat Jul 27 10:18:46 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Sat, 27 Jul 2013 13:18:46 -0400 Subject: Stability of lambda serialization In-Reply-To: References: <286109594.871872.1374598513380.JavaMail.root@redhat.com> <572125674.885842.1374600926640.JavaMail.root@redhat.com> Message-ID: <51F400F6.3030008@oracle.com> Yes, when this came up the first several times, the consensus of the EG was that we would build on the de-facto, practical constraint that people have learned to live with for years when dealing with serialization in the presence of inner classes: make sure the same classfile bits are on both sides of the pipe. Inner classes have a naming instability, where you have a stable method name but an unstable class name. Lambdas have the opposite, but effectively equivalent instability: a stable class name but an unstable method name. People have learned to use serialization with inner classes by the technique of keeping the classfile bits the same on both sides of the wire. The exact same technique enables serialization of lambdas. In particular, the EG rejected the extremes of "make it perfect" and "don't allow it at all" as being "the perfect is the enemy of the good." Inner classes have been exactly this hostile to serialization for 15 years, but people who want to use serialization have learned coping techniques to do so. The compromises in the middle are squishy but better than either extreme. Of course, if we can make it better without exposing additional complexity, so much the better. For example, we did a few rounds of this by encoding the hash of the method name and signature so that simply reordering the methods in a class did not trigger such instabilities. But the goal was never perfect stability; it was just "at least as good as inner classes." I believe we've reached that goal. Do you have a concrete proposal for how we would improve lambda serialization to be robust in the face of changes in order of capture? On the surface it looks entirely possible. On 7/27/2013 12:52 PM, Sam Pullara wrote: > I think the requirement should be that you have the same classes on each > side of the wire. > > Sam > > > On Tue, Jul 23, 2013 at 10:35 AM, Scott Stark > wrote: > > Red Hat has a concern regarding how fragile the default > serialization behavior of lambda expressions is in the current > reference implementation, currently: > ironmaiden:OpenJDK starksm$ > /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java -version > java version "1.8.0-ea" > Java(TM) SE Runtime Environment (build 1.8.0-ea-b98) > Java HotSpot(TM) 64-Bit Server VM (build 25.0-b40, mixed mode) > > The problem is that the serialized form of a lambda expression > depends on the order in which captured arguments are declared. The > attached simple example demonstrates how easy it is for a trivial > reordering of the lambda code block to result in an inability to > deserialize a previously saved expression. > > To produce this exception: > 1. Run the serialization.AuthenticationContext.testWriteLambda > method with the lambda expression written as: > Authenticator a = (Authenticator & Serializable) (String > principal, char[] pass) -> { > // Run with p declared first when writing out the > /tmp/testWriteLambda.bin, then switch > // to declare u first when running testReadLambda > String p = "-> Password " + password + " <-"; > String u = "-> User " + user + " <-"; > return u + " " + p; > }; > 2. Change the lambda expression to: > Authenticator a = (Authenticator & Serializable) (String > principal, char[] pass) -> { > // Run with p declared first when writing out the > /tmp/testWriteLambda.bin, then switch > // to declare u first when running testReadLambda > String u = "-> User " + user + " <-"; > String p = "-> Password " + password + " <-"; > return u + " " + p; > }; > > Recompile and run serialization.AuthenticationContext.testReadLambda > to produce: > > java.io.IOException: unexpected exception type > at > java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538) > at > java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1110) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) > at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > serialization.AuthenticationContext.testReadLambda(AuthenticationContext.java:34) > ... > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at > java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:222) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at > java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104) > ... 30 more > Caused by: java.lang.IllegalArgumentException: Invalid lambda > deserialization > at > serialization.AuthenticationContext.$deserializeLambda$(AuthenticationContext.java:1) > ... 40 more > > One does not see the same level of sensitivity to the ordering of > the serialization fields in a POJO as demonstrated by the > serialization.AuthenticationContext.testWritePOJO/testReadPOJO cases > where one can reorder the TestPOJO.{user,password} fields without > having serialization fail. > > We would like to see at least that level of stability of the > serialized form of lambda expressions. > > From forax at univ-mlv.fr Sat Jul 27 10:36:06 2013 From: forax at univ-mlv.fr (Remi Forax) Date: Sat, 27 Jul 2013 19:36:06 +0200 Subject: Stability of lambda serialization In-Reply-To: <51F400F6.3030008@oracle.com> References: <286109594.871872.1374598513380.JavaMail.root@redhat.com> <572125674.885842.1374600926640.JavaMail.root@redhat.com> <51F400F6.3030008@oracle.com> Message-ID: <51F40506.7070407@univ-mlv.fr> On 07/27/2013 07:18 PM, Brian Goetz wrote: > Yes, when this came up the first several times, the consensus of the > EG was that we would build on the de-facto, practical constraint that > people have learned to live with for years when dealing with > serialization in the presence of inner classes: make sure the same > classfile bits are on both sides of the pipe. > > Inner classes have a naming instability, where you have a stable > method name but an unstable class name. Lambdas have the opposite, > but effectively equivalent instability: a stable class name but an > unstable method name. People have learned to use serialization with > inner classes by the technique of keeping the classfile bits the same > on both sides of the wire. The exact same technique enables > serialization of lambdas. > > In particular, the EG rejected the extremes of "make it perfect" and > "don't allow it at all" as being "the perfect is the enemy of the > good." Inner classes have been exactly this hostile to serialization > for 15 years, but people who want to use serialization have learned > coping techniques to do so. The compromises in the middle are squishy > but better than either extreme. > > Of course, if we can make it better without exposing additional > complexity, so much the better. For example, we did a few rounds of > this by encoding the hash of the method name and signature so that > simply reordering the methods in a class did not trigger such > instabilities. But the goal was never perfect stability; it was just > "at least as good as inner classes." I believe we've reached that goal. > > Do you have a concrete proposal for how we would improve lambda > serialization to be robust in the face of changes in order of capture? > On the surface it looks entirely possible. First, I hope there is not a hashtable somewhere in javac and that the order of capture is really defined by the order of the captured variables inside the body of the lambda. Also, I find this order better than the order of the declared captured variable in the outer scopes. The only solution I see for that is to also encodes the name of the captured variables (we do something like this for enums) and to encode the name of the parameter of the generated method lambda$$$$. It will make the serialized binary form bigger and the deserialization (a little, we can leverage invokedynamic here to do the shuffling) slower. And anyway, if one local variable is renamed it will not worked. I don't know if we want to go that far ... cheers, R?mi > > On 7/27/2013 12:52 PM, Sam Pullara wrote: >> I think the requirement should be that you have the same classes on each >> side of the wire. >> >> Sam >> >> >> On Tue, Jul 23, 2013 at 10:35 AM, Scott Stark > > wrote: >> >> Red Hat has a concern regarding how fragile the default >> serialization behavior of lambda expressions is in the current >> reference implementation, currently: >> ironmaiden:OpenJDK starksm$ >> /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java >> -version >> java version "1.8.0-ea" >> Java(TM) SE Runtime Environment (build 1.8.0-ea-b98) >> Java HotSpot(TM) 64-Bit Server VM (build 25.0-b40, mixed mode) >> >> The problem is that the serialized form of a lambda expression >> depends on the order in which captured arguments are declared. The >> attached simple example demonstrates how easy it is for a trivial >> reordering of the lambda code block to result in an inability to >> deserialize a previously saved expression. >> >> To produce this exception: >> 1. Run the serialization.AuthenticationContext.testWriteLambda >> method with the lambda expression written as: >> Authenticator a = (Authenticator & Serializable) (String >> principal, char[] pass) -> { >> // Run with p declared first when writing out the >> /tmp/testWriteLambda.bin, then switch >> // to declare u first when running testReadLambda >> String p = "-> Password " + password + " <-"; >> String u = "-> User " + user + " <-"; >> return u + " " + p; >> }; >> 2. Change the lambda expression to: >> Authenticator a = (Authenticator & Serializable) (String >> principal, char[] pass) -> { >> // Run with p declared first when writing out the >> /tmp/testWriteLambda.bin, then switch >> // to declare u first when running testReadLambda >> String u = "-> User " + user + " <-"; >> String p = "-> Password " + password + " <-"; >> return u + " " + p; >> }; >> >> Recompile and run serialization.AuthenticationContext.testReadLambda >> to produce: >> >> java.io.IOException: unexpected exception type >> at >> java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538) >> at >> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1110) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >> at >> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >> at >> serialization.AuthenticationContext.testReadLambda(AuthenticationContext.java:34) >> ... >> Caused by: java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at >> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:222) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at >> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104) >> ... 30 more >> Caused by: java.lang.IllegalArgumentException: Invalid lambda >> deserialization >> at >> serialization.AuthenticationContext.$deserializeLambda$(AuthenticationContext.java:1) >> ... 40 more >> >> One does not see the same level of sensitivity to the ordering of >> the serialization fields in a POJO as demonstrated by the >> serialization.AuthenticationContext.testWritePOJO/testReadPOJO cases >> where one can reorder the TestPOJO.{user,password} fields without >> having serialization fail. >> >> We would like to see at least that level of stability of the >> serialized form of lambda expressions. >> >> From david.lloyd at redhat.com Mon Jul 29 17:40:48 2013 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 29 Jul 2013 19:40:48 -0500 Subject: Stability of lambda serialization In-Reply-To: <51F400F6.3030008@oracle.com> References: <286109594.871872.1374598513380.JavaMail.root@redhat.com> <572125674.885842.1374600926640.JavaMail.root@redhat.com> <51F400F6.3030008@oracle.com> Message-ID: <51F70B90.1030900@redhat.com> As before, I continue to strongly disagree with this so-called "de facto" constraint. In reality, we see real users who rely heavily on the *real* constraints which are outlined in the specification of serialization. I think that real-world users are going to encounter bizarre issues at best, and security vulnerabilities at worst, due to the behavior of capture. As I always do when this particular truism is brought up, I will counter your "the perfect is the enemy of the good" with "the good is the enemy of the correct". A clever saying is never enough to justify incorrect behavior. I think it's perfectly reasonable to capture "this", at most, but beyond that I think there is no benefit to having capture, at all, in the face of its instability. I think the "equivalent stability" argument is also a bit fallacious. It's saying "anonymous classes are bad, so that's a free pass to do this similarly badly as well, in whatever way we want". To be "as good as anonymous classes", we'd have to have capture which is stable by name at least. But I personally I think this criteria is arbitrary and unnecessary in any case. I think that we can easily guarantee stability of both captured variables and of method resolution by only allowing serialization of named method references and forbidding capture (other than "this"). On 07/27/2013 12:18 PM, Brian Goetz wrote: > Yes, when this came up the first several times, the consensus of the EG > was that we would build on the de-facto, practical constraint that > people have learned to live with for years when dealing with > serialization in the presence of inner classes: make sure the same > classfile bits are on both sides of the pipe. > > Inner classes have a naming instability, where you have a stable method > name but an unstable class name. Lambdas have the opposite, but > effectively equivalent instability: a stable class name but an unstable > method name. People have learned to use serialization with inner > classes by the technique of keeping the classfile bits the same on both > sides of the wire. The exact same technique enables serialization of > lambdas. > > In particular, the EG rejected the extremes of "make it perfect" and > "don't allow it at all" as being "the perfect is the enemy of the good." > Inner classes have been exactly this hostile to serialization for 15 > years, but people who want to use serialization have learned coping > techniques to do so. The compromises in the middle are squishy but > better than either extreme. > > Of course, if we can make it better without exposing additional > complexity, so much the better. For example, we did a few rounds of > this by encoding the hash of the method name and signature so that > simply reordering the methods in a class did not trigger such > instabilities. But the goal was never perfect stability; it was just > "at least as good as inner classes." I believe we've reached that goal. > > Do you have a concrete proposal for how we would improve lambda > serialization to be robust in the face of changes in order of capture? > On the surface it looks entirely possible. > > On 7/27/2013 12:52 PM, Sam Pullara wrote: >> I think the requirement should be that you have the same classes on each >> side of the wire. >> >> Sam >> >> >> On Tue, Jul 23, 2013 at 10:35 AM, Scott Stark > > wrote: >> >> Red Hat has a concern regarding how fragile the default >> serialization behavior of lambda expressions is in the current >> reference implementation, currently: >> ironmaiden:OpenJDK starksm$ >> >> /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java >> -version >> java version "1.8.0-ea" >> Java(TM) SE Runtime Environment (build 1.8.0-ea-b98) >> Java HotSpot(TM) 64-Bit Server VM (build 25.0-b40, mixed mode) >> >> The problem is that the serialized form of a lambda expression >> depends on the order in which captured arguments are declared. The >> attached simple example demonstrates how easy it is for a trivial >> reordering of the lambda code block to result in an inability to >> deserialize a previously saved expression. >> >> To produce this exception: >> 1. Run the serialization.AuthenticationContext.testWriteLambda >> method with the lambda expression written as: >> Authenticator a = (Authenticator & Serializable) (String >> principal, char[] pass) -> { >> // Run with p declared first when writing out the >> /tmp/testWriteLambda.bin, then switch >> // to declare u first when running testReadLambda >> String p = "-> Password " + password + " <-"; >> String u = "-> User " + user + " <-"; >> return u + " " + p; >> }; >> 2. Change the lambda expression to: >> Authenticator a = (Authenticator & Serializable) (String >> principal, char[] pass) -> { >> // Run with p declared first when writing out the >> /tmp/testWriteLambda.bin, then switch >> // to declare u first when running testReadLambda >> String u = "-> User " + user + " <-"; >> String p = "-> Password " + password + " <-"; >> return u + " " + p; >> }; >> >> Recompile and run serialization.AuthenticationContext.testReadLambda >> to produce: >> >> java.io.IOException: unexpected exception type >> at >> >> java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538) >> at >> >> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1110) >> at >> >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >> at >> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >> at >> >> serialization.AuthenticationContext.testReadLambda(AuthenticationContext.java:34) >> >> ... >> Caused by: java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at >> >> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:222) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at >> >> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104) >> ... 30 more >> Caused by: java.lang.IllegalArgumentException: Invalid lambda >> deserialization >> at >> >> serialization.AuthenticationContext.$deserializeLambda$(AuthenticationContext.java:1) >> >> ... 40 more >> >> One does not see the same level of sensitivity to the ordering of >> the serialization fields in a POJO as demonstrated by the >> serialization.AuthenticationContext.testWritePOJO/testReadPOJO cases >> where one can reorder the TestPOJO.{user,password} fields without >> having serialization fail. >> >> We would like to see at least that level of stability of the >> serialized form of lambda expressions. >> >> -- - DML From david.lloyd at redhat.com Tue Jul 30 06:15:57 2013 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 30 Jul 2013 08:15:57 -0500 Subject: Stability of lambda serialization In-Reply-To: <51F70B90.1030900@redhat.com> References: <286109594.871872.1374598513380.JavaMail.root@redhat.com> <572125674.885842.1374600926640.JavaMail.root@redhat.com> <51F400F6.3030008@oracle.com> <51F70B90.1030900@redhat.com> Message-ID: <51F7BC8D.5040401@redhat.com> I realize I was ambiguous: On 07/29/2013 07:40 PM, David M. Lloyd wrote: > As before, I continue to strongly disagree with this so-called "de > facto" constraint. In reality, we see real users who rely heavily on > the *real* constraints which are outlined in the specification of > serialization. I think that real-world users are going to encounter > bizarre issues at best, and security vulnerabilities at worst, due to > the behavior of capture. ...due to the behavior of capture with respect to serialization. > As I always do when this particular truism is brought up, I will counter > your "the perfect is the enemy of the good" with "the good is the enemy > of the correct". A clever saying is never enough to justify incorrect > behavior. I think it's perfectly reasonable to capture "this", at most, > but beyond that I think there is no benefit to having capture, at all, > in the face of its instability. ...having capture in the context of serialization... > I think the "equivalent stability" argument is also a bit fallacious. > It's saying "anonymous classes are bad, so that's a free pass to do this > similarly badly as well, in whatever way we want". To be "as good as > anonymous classes", we'd have to have capture which is stable by name at > least. ...capture in the context of serialization... > But I personally I think this criteria is arbitrary and unnecessary in > any case. I think that we can easily guarantee stability of both > captured variables and of method resolution by only allowing > serialization of named method references and forbidding capture (other > than "this"). ...forbidding capture in serializable lambdas... > On 07/27/2013 12:18 PM, Brian Goetz wrote: >> Yes, when this came up the first several times, the consensus of the EG >> was that we would build on the de-facto, practical constraint that >> people have learned to live with for years when dealing with >> serialization in the presence of inner classes: make sure the same >> classfile bits are on both sides of the pipe. >> >> Inner classes have a naming instability, where you have a stable method >> name but an unstable class name. Lambdas have the opposite, but >> effectively equivalent instability: a stable class name but an unstable >> method name. People have learned to use serialization with inner >> classes by the technique of keeping the classfile bits the same on both >> sides of the wire. The exact same technique enables serialization of >> lambdas. >> >> In particular, the EG rejected the extremes of "make it perfect" and >> "don't allow it at all" as being "the perfect is the enemy of the good." >> Inner classes have been exactly this hostile to serialization for 15 >> years, but people who want to use serialization have learned coping >> techniques to do so. The compromises in the middle are squishy but >> better than either extreme. >> >> Of course, if we can make it better without exposing additional >> complexity, so much the better. For example, we did a few rounds of >> this by encoding the hash of the method name and signature so that >> simply reordering the methods in a class did not trigger such >> instabilities. But the goal was never perfect stability; it was just >> "at least as good as inner classes." I believe we've reached that goal. >> >> Do you have a concrete proposal for how we would improve lambda >> serialization to be robust in the face of changes in order of capture? >> On the surface it looks entirely possible. >> >> On 7/27/2013 12:52 PM, Sam Pullara wrote: >>> I think the requirement should be that you have the same classes on each >>> side of the wire. >>> >>> Sam >>> >>> >>> On Tue, Jul 23, 2013 at 10:35 AM, Scott Stark >> > wrote: >>> >>> Red Hat has a concern regarding how fragile the default >>> serialization behavior of lambda expressions is in the current >>> reference implementation, currently: >>> ironmaiden:OpenJDK starksm$ >>> >>> /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java >>> -version >>> java version "1.8.0-ea" >>> Java(TM) SE Runtime Environment (build 1.8.0-ea-b98) >>> Java HotSpot(TM) 64-Bit Server VM (build 25.0-b40, mixed mode) >>> >>> The problem is that the serialized form of a lambda expression >>> depends on the order in which captured arguments are declared. The >>> attached simple example demonstrates how easy it is for a trivial >>> reordering of the lambda code block to result in an inability to >>> deserialize a previously saved expression. >>> >>> To produce this exception: >>> 1. Run the serialization.AuthenticationContext.testWriteLambda >>> method with the lambda expression written as: >>> Authenticator a = (Authenticator & Serializable) (String >>> principal, char[] pass) -> { >>> // Run with p declared first when writing out the >>> /tmp/testWriteLambda.bin, then switch >>> // to declare u first when running testReadLambda >>> String p = "-> Password " + password + " <-"; >>> String u = "-> User " + user + " <-"; >>> return u + " " + p; >>> }; >>> 2. Change the lambda expression to: >>> Authenticator a = (Authenticator & Serializable) (String >>> principal, char[] pass) -> { >>> // Run with p declared first when writing out the >>> /tmp/testWriteLambda.bin, then switch >>> // to declare u first when running testReadLambda >>> String u = "-> User " + user + " <-"; >>> String p = "-> Password " + password + " <-"; >>> return u + " " + p; >>> }; >>> >>> Recompile and run serialization.AuthenticationContext.testReadLambda >>> to produce: >>> >>> java.io.IOException: unexpected exception type >>> at >>> >>> java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538) >>> >>> at >>> >>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1110) >>> at >>> >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1807) >>> >>> at >>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) >>> at >>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >>> at >>> >>> serialization.AuthenticationContext.testReadLambda(AuthenticationContext.java:34) >>> >>> >>> ... >>> Caused by: java.lang.reflect.InvocationTargetException >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> at >>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> >>> >>> at >>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> >>> at >>> >>> java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:222) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> at >>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> >>> >>> at >>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> >>> at >>> >>> java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1104) >>> ... 30 more >>> Caused by: java.lang.IllegalArgumentException: Invalid lambda >>> deserialization >>> at >>> >>> serialization.AuthenticationContext.$deserializeLambda$(AuthenticationContext.java:1) >>> >>> >>> ... 40 more >>> >>> One does not see the same level of sensitivity to the ordering of >>> the serialization fields in a POJO as demonstrated by the >>> serialization.AuthenticationContext.testWritePOJO/testReadPOJO cases >>> where one can reorder the TestPOJO.{user,password} fields without >>> having serialization fail. >>> >>> We would like to see at least that level of stability of the >>> serialized form of lambda expressions. >>> >>> > > -- - DML From brian.goetz at oracle.com Tue Jul 30 10:27:30 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 30 Jul 2013 10:27:30 -0700 Subject: EG meeting In-Reply-To: References: <51CCDA17.70605@oracle.com> Message-ID: Reminder: Meeting on Thursday 9A. I have the following attendees down: Oracle: Brian, Dan, Maurizio, Robert, Joe IBM: Dan and Ryan Google: Kevin and Colin Jetbrains: Andrey and Anna Joe B Sam Remi Vlad Crazy Bob I think this is close to the capacity of the room we reserved, so if I've missed anyone, let me know and I'll see what I can do. > > The plans are finalized, we will have an in-person meeting the Thursday of JVM Language Summit week, 9:00AM at "The Mansion" on Oracle's Santa Clara campus (same place as last year). From spullara at gmail.com Wed Jul 31 09:20:58 2013 From: spullara at gmail.com (Sam Pullara) Date: Wed, 31 Jul 2013 09:20:58 -0700 Subject: EG meeting In-Reply-To: References: <51CCDA17.70605@oracle.com> Message-ID: <2B48B275-0FAC-4A07-8A42-44C79C8B2200@gmail.com> Unfortunately, I have to bow out of this. I failed to reserve it on my calendar and now it is overrun. Sorry, Sam On Jul 30, 2013, at 10:27 AM, Brian Goetz wrote: > Reminder: Meeting on Thursday 9A. > > I have the following attendees down: > > Oracle: Brian, Dan, Maurizio, Robert, Joe > IBM: Dan and Ryan > Google: Kevin and Colin > Jetbrains: Andrey and Anna > Joe B > Sam > Remi > Vlad > Crazy Bob > > I think this is close to the capacity of the room we reserved, so if I've missed anyone, let me know and I'll see what I can do. > >>> The plans are finalized, we will have an in-person meeting the Thursday of JVM Language Summit week, 9:00AM at "The Mansion" on Oracle's Santa Clara campus (same place as last year). >