From ali.ebrahimi1781 at gmail.com Sat Mar 1 02:17:52 2014 From: ali.ebrahimi1781 at gmail.com (Ali Ebrahimi) Date: Sat, 1 Mar 2014 13:47:52 +0330 Subject: Signature of MethodHandleInfo.reflectAs is not specific enough In-Reply-To: <11925E47-E9C3-4234-ADE5-0599CC71254B@oracle.com> References: <528008C9.7000905@univ-mlv.fr> <11925E47-E9C3-4234-ADE5-0599CC71254B@oracle.com> Message-ID: Hi, On Sat, Mar 1, 2014 at 12:16 AM, John Rose wrote: > On Feb 25, 2014, at 3:13 AM, Ali Ebrahimi > wrote: > > I know, this is too late, but I want to share my suggestion: > > public T reflectAs(Class super T> expected, MethodHandles.Lookup lookup) > > > Isn't this the same as > > public T reflectAs... > Oh, sorry, this is my bad. I mean this: public T reflectAs(Class expected, MethodHandles.Lookup lookup) > > I think we considered AccessibleObject but rejected it as not buying > anything significant compared with Member which is an interface. > > Perhaps > > public T reflectAs... > I considered this case, but unfortunately compiler accepts following test case with this signature: String ss = reflectAs(String.class, MethodHandles.lookup()); //OK!!!!!!! This seams as compiler bug, any way with compiler catches error. Finally, one case that compiler accepts with both signature is: Object mo = reflectAs(Object.class, MethodHandles.lookup()); Maybe I'm wrong for this, Dan Smith can better interpret this. T inferred to Object and T is not within its bound (AccessibleObject&Member). Regards, Ali Ebrahimi From timo.kinnunen at gmail.com Sun Mar 2 06:08:47 2014 From: timo.kinnunen at gmail.com (=?utf-8?Q?Timo_Kinnunen?=) Date: Sun, 2 Mar 2014 14:08:47 +0000 Subject: =?utf-8?Q?Woe_is_me_for_using_=3F_super_T_in_a_method_call_(part_1)?= Message-ID: <53136a1d.83e00e0a.43b2.fffffe00@mx.google.com> Hi, These examples used to compile in ECJ until recently and I didn?t bat an eyelid at them, but not anymore. Looks like they haven?t compiled in javac for maybe lot longer. The first one is just precious in its simplicity: private static void internalTake(Consumer[] c) {} public static void take(Consumer[] c) { internalTake(c);//error? } Wow, WTF? Just pick one super-type of T, any one! Here?s a more fully-fledged motivating example: private static void callerOfTake2() { take2( "A test string", asArray( (CharSequence s) -> System.out.println(s), (String s) -> System.out.println("S" + s), (Object s) -> System.out.println("O" + s))); } @SafeVarargs private static Consumer[] asArray(Consumer... cees) { return cees; } static void take2(T t, Consumer[] c) { internalTake2(t, c);//error? } private static void internalTake2(SUB_E e, Consumer[] c) { for(Consumer consumer : c) { consumer.accept(e); } } So what?s going on here? Javac?s explanation doesn?t really explain where the disconnect is between my understanding of how these wildcards should work and how they actually work. I?m splitting the other example to another email because this is getting long enough already. -- Have a nice day, Timo. Sent from Windows Mail From timo.kinnunen at gmail.com Sun Mar 2 09:27:57 2014 From: timo.kinnunen at gmail.com (=?utf-8?Q?Timo_Kinnunen?=) Date: Sun, 2 Mar 2014 17:27:57 +0000 Subject: =?utf-8?Q?Woe_is_me_for_using_=3F_super_T_in_a_method_call_(part_2)?= Message-ID: <53136e4b.c5630e0a.4725.2f6f@mx.google.com> Hi, Picking up from part 1, here?s the other, related example: public class RegressionsMaybe { static void typeParametersMadeRelatedForDemoPurposes() { BinaryOperator> op1_1 = Consumer::andThen;//not error? BinaryOperator> op2_1 = Consumer::andThen;//ok BinaryOperator> op1_2 = RegressionsMaybe::andThen1;//error? BinaryOperator> op2_2 = RegressionsMaybe::andThen1;//ok.. BinaryOperator> op1_3 = RegressionsMaybe::andThen2;//error? BinaryOperator> op2_3 = RegressionsMaybe::andThen2;//ok.. } private static Consumer andThen1(Consumer before, Consumer after) { return (T t) -> { before.accept(t); after.accept(t); }; } private static Consumer andThen2(Consumer before, Consumer after) { return (T t) -> { before.accept(t); after.accept(t); }; } } In the above, ECJ shows 3 errors but javac only shows 2. Before I report the op1_1 line not compiling in ECJ as a bug I?d like to understand why it shouldn?t be an error. Also why the other ones are or are not errors. Yes, I?d really like to understand all of it. -- Have a nice day, Timo. Sent from Windows Mail From zhong.j.yu at gmail.com Sun Mar 2 21:53:06 2014 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Sun, 2 Mar 2014 23:53:06 -0600 Subject: Woe is me for using ? super T in a method call (part 1) In-Reply-To: <53136a1d.83e00e0a.43b2.fffffe00@mx.google.com> References: <53136a1d.83e00e0a.43b2.fffffe00@mx.google.com> Message-ID: Wildcard capture works on the 1st level wildcard: a List object is a List, for some x that x<:String It does not work on deeper levels: a List> object is NOT a List> for some x. In your example, we'll need Consumer <: Consumer no choice of E can make that happen. The solution is to almost always use wildcard on co/contra-variant types: internalTake(Consumer[] c) The problem of the solution is that our code is litterred with wildcards (in a very ugly notation no less) Zhong Yu On Sun, Mar 2, 2014 at 8:08 AM, Timo Kinnunen wrote: > Hi, > > > > > These examples used to compile in ECJ until recently and I didn't bat an eyelid at them, but not anymore. Looks like they haven't compiled in javac for maybe lot longer. > > > > > The first one is just precious in its simplicity: > > > > > private static void internalTake(Consumer[] c) {} > > public static void take(Consumer[] c) { > > internalTake(c);//error? > > } > > > > > Wow, WTF? Just pick one super-type of T, any one! > > > > > Here's a more fully-fledged motivating example: > > > > > private static void callerOfTake2() { > > take2( > > "A test string", > > asArray( > > (CharSequence s) -> System.out.println(s), > > (String s) -> System.out.println("S" + s), > > (Object s) -> System.out.println("O" + s))); > > } > > @SafeVarargs > > private static Consumer[] asArray(Consumer... cees) { > > return cees; > > } > > static void take2(T t, Consumer[] c) { > > internalTake2(t, c);//error? > > } > > private static void internalTake2(SUB_E e, Consumer[] c) { > > for(Consumer consumer : c) { > > consumer.accept(e); > > } > > } > > > > > > So what's going on here? Javac's explanation doesn't really explain where the disconnect is between my understanding of how these wildcards should work and how they actually work. > > > I'm splitting the other example to another email because this is getting long enough already. > > > -- > Have a nice day, > Timo. > > Sent from Windows Mail > From brian.goetz at oracle.com Mon Mar 3 13:36:28 2014 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 03 Mar 2014 16:36:28 -0500 Subject: JEP 186: Collection Literals In-Reply-To: References: Message-ID: <5314F5DC.3070804@oracle.com> Just to close the loop here, we've reached the conclusion that we will not be pursuing collection literals as a language feature right now, but instead proceeding in the short term with a library-based proposal that is similar to what Stephen has described here, with a few ideas from Guava's ImmutableXxx classes. Our motivation is largely as outlined earlier in this thread: - The "simple" version of this feature (sets, lists, maps only) is not very satisfying or popular; the "extensible" version of this feature is open-ended, messy, and virtually guaranteed to way overrun its design budget; - The library-based version gives us X% of the benefit for 1% of the cost, where X >> 1; - Value types are coming, and the "what would this feature look like" in a world with value types may well be quite different than in a world without, suggesting it would be questionable to try and do this work before value types; - We are better off focusing our language-design bandwidth on addressing more the foundational issues underlying a library-based version (including: more efficient varargs, array constants in the constant pool, immutable arrays, and support for caching (and reclaiming under pressure) intermediate immutable results). Separately, I'll try and write up my notes about the various places this exploration led us. On 2/19/2014 1:02 PM, Stephen Colebourne wrote: > On 21 January 2014 19:39, Brian Goetz wrote: >> 3. The real pain is in Maps, not Lists, Sets, or Arrays. Library-based >> solutions would be mostly acceptable for the latter, but we still lack a >> reasonable way to describe pair literals, which is what's in the way of >> a library-based solution for Map (other than a MapBuilder.) Static >> methods in interfaces make a library-based solution more practical. >> With value types, though, library-based solutions for Map become far >> more practical too. > > Just to note on this, I previously sent a proposal to core-libs: > http://mail.openjdk.java.net/pipermail/lambda-dev/2013-June/010228.html > and a proof of concept patch: > https://gist.github.com/jodastephen/6659515 > > The heart of the concept was static methods on interfaces: > Collection.empty() > Collection.of(T...) > List.empty() > List.of(T...) > Set.empty() > Set.of(T...) > Map.empty() > Map.of(Entry...) > Map.Entry.of(K, V) > > > Each of these methods would return immutable implementations. There is > a case for extending the methods to Iterator and other collection > types, however these are the most important. These follow the designs > of Stream static methods IIRC. > > I think a complete solution would involve matching static methods on > the collection classes as well, eg > ArrayList.of(T...) > TreeSet.of(T...) > > I note that the Map variant in the patch is a cunning way to beat the > absence of a pair/tuple and generics, but it would require some > thought before adoption in the JDK. > > Stephen > From paulo.silveira at caelum.com.br Tue Mar 4 12:13:18 2014 From: paulo.silveira at caelum.com.br (Paulo Silveira) Date: Tue, 4 Mar 2014 17:13:18 -0300 Subject: javadoc small error for Optional.map Message-ID: We have a simple example for Optional.map: Optional fis = names.stream().filter(name -> !isProcessedYet(name)) .findFirst() .map(name -> new FileInputStream(name)); But it would never compile, since FileInputStream::new throws IOException and Function::apply does not. -- Paulo Silveira www.caelum.com.br www.casadocodigo.com.br www.alura.com.br From forax at univ-mlv.fr Thu Mar 6 03:29:49 2014 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 06 Mar 2014 12:29:49 +0100 Subject: Lambda and multi-catch Message-ID: <53185C2D.4090907@univ-mlv.fr> A colleague of mine have find a hairy bug, here is a simple code to reproduce it. import java.io.IOException; public class FunWithMultiCatch { public static void main(String[] args) { Runnable r = () -> { try { Object o = null; o.getClass(); throw new IOException(); } catch(IOException | IllegalArgumentException e) { System.out.println("KO !"); } catch(RuntimeException e) { System.out.println("OK !"); } }; r.run(); } } It prints 'KO !' :( The problem can be reproduced for any multi-catch inside a lambda, the exception table is clearly wrong, the IOException and IAE are merged to their common supertype (java.lang.Exception). private static void lambda$main$0(); Code: 0: aconst_null ... 14: athrow 15: astore_0 16: getstatic #8 // Field java/lang/System.out:Ljava/io/PrintStream; 19: ldc #9 // String OK ! 21: invokevirtual #10 // Method java/io/PrintStream.println:(Ljava/lang/String;)V 24: goto 36 27: astore_0 28: getstatic #8 // Field java/lang/System.out:Ljava/io/PrintStream; 31: ldc #12 // String KO ! 33: invokevirtual #10 // Method java/io/PrintStream.println:(Ljava/lang/String;)V 36: return Exception table: from to target type 0 15 15 Class java/lang/Exception <------------ Oops 0 15 27 Class java/lang/RuntimeException The current workaround is to not use a multi-catch in a lambda. It seems that some version of Eclipse had the same issue but it's fixed with the one I use (from 21 feb). cheers, R?mi From david.holmes at oracle.com Thu Mar 6 21:15:30 2014 From: david.holmes at oracle.com (David Holmes) Date: Fri, 07 Mar 2014 15:15:30 +1000 Subject: Lambda and multi-catch In-Reply-To: <53185C2D.4090907@univ-mlv.fr> References: <53185C2D.4090907@univ-mlv.fr> Message-ID: <531955F2.1010602@oracle.com> On 6/03/2014 9:29 PM, Remi Forax wrote: > A colleague of mine have find a hairy bug, > here is a simple code to reproduce it. > > import java.io.IOException; > > public class FunWithMultiCatch { > public static void main(String[] args) { > Runnable r = () -> { > try { > Object o = null; > o.getClass(); > throw new IOException(); > } catch(IOException | IllegalArgumentException e) { > System.out.println("KO !"); > } catch(RuntimeException e) { > System.out.println("OK !"); > } > }; > r.run(); > } > } > > It prints 'KO !' :( What am I missing? We throw IOException and have a catch block for IOException so I would expect it to print "KO !". I'm more surprised that a non-lambda version prints "OK !" > The problem can be reproduced for any multi-catch inside a lambda, > the exception table is clearly wrong, the IOException and IAE are merged > to their common supertype (java.lang.Exception). That would definitely seem wrong, but more of an issue with catching RuntimeExceptions in the first block that should be handled by the second catch block. eg change it to throw IllegalStateException() instead: if (false) throw new IOException(); throw new IllegalStateException(); -> "KO !" !!!WT! David ----- > private static void lambda$main$0(); > Code: > 0: aconst_null > ... > 14: athrow > 15: astore_0 > 16: getstatic #8 // Field > java/lang/System.out:Ljava/io/PrintStream; > 19: ldc #9 // String OK ! > 21: invokevirtual #10 // Method > java/io/PrintStream.println:(Ljava/lang/String;)V > 24: goto 36 > 27: astore_0 > 28: getstatic #8 // Field > java/lang/System.out:Ljava/io/PrintStream; > 31: ldc #12 // String KO ! > 33: invokevirtual #10 // Method > java/io/PrintStream.println:(Ljava/lang/String;)V > 36: return > Exception table: > from to target type > 0 15 15 Class > java/lang/Exception <------------ Oops > 0 15 27 Class java/lang/RuntimeException > > The current workaround is to not use a multi-catch in a lambda. > > It seems that some version of Eclipse had the same issue but it's fixed > with the one I use (from 21 feb). > > cheers, > R?mi > > From sam at sampullara.com Thu Mar 6 21:53:55 2014 From: sam at sampullara.com (Sam Pullara) Date: Thu, 6 Mar 2014 21:53:55 -0800 Subject: Lambda and multi-catch In-Reply-To: <531955F2.1010602@oracle.com> References: <53185C2D.4090907@univ-mlv.fr> <531955F2.1010602@oracle.com> Message-ID: It is throwing an NPE when he dereferences the null o -- I thought the same thing the first time I looked at it. Sam On Thu, Mar 6, 2014 at 9:15 PM, David Holmes wrote: > On 6/03/2014 9:29 PM, Remi Forax wrote: > > A colleague of mine have find a hairy bug, > > here is a simple code to reproduce it. > > > > import java.io.IOException; > > > > public class FunWithMultiCatch { > > public static void main(String[] args) { > > Runnable r = () -> { > > try { > > Object o = null; > > o.getClass(); > > throw new IOException(); > > } catch(IOException | IllegalArgumentException e) { > > System.out.println("KO !"); > > } catch(RuntimeException e) { > > System.out.println("OK !"); > > } > > }; > > r.run(); > > } > > } > > > > It prints 'KO !' :( > > What am I missing? We throw IOException and have a catch block for > IOException so I would expect it to print "KO !". > > I'm more surprised that a non-lambda version prints "OK !" > > > The problem can be reproduced for any multi-catch inside a lambda, > > the exception table is clearly wrong, the IOException and IAE are merged > > to their common supertype (java.lang.Exception). > > That would definitely seem wrong, but more of an issue with catching > RuntimeExceptions in the first block that should be handled by the > second catch block. eg change it to throw IllegalStateException() instead: > > if (false) throw new IOException(); > throw new IllegalStateException(); > > -> "KO !" !!!WT! > > David > ----- > > > private static void lambda$main$0(); > > Code: > > 0: aconst_null > > ... > > 14: athrow > > 15: astore_0 > > 16: getstatic #8 // Field > > java/lang/System.out:Ljava/io/PrintStream; > > 19: ldc #9 // String OK ! > > 21: invokevirtual #10 // Method > > java/io/PrintStream.println:(Ljava/lang/String;)V > > 24: goto 36 > > 27: astore_0 > > 28: getstatic #8 // Field > > java/lang/System.out:Ljava/io/PrintStream; > > 31: ldc #12 // String KO ! > > 33: invokevirtual #10 // Method > > java/io/PrintStream.println:(Ljava/lang/String;)V > > 36: return > > Exception table: > > from to target type > > 0 15 15 Class > > java/lang/Exception <------------ Oops > > 0 15 27 Class java/lang/RuntimeException > > > > The current workaround is to not use a multi-catch in a lambda. > > > > It seems that some version of Eclipse had the same issue but it's fixed > > with the one I use (from 21 feb). > > > > cheers, > > R?mi > > > > > > From david.holmes at oracle.com Thu Mar 6 22:15:38 2014 From: david.holmes at oracle.com (David Holmes) Date: Fri, 07 Mar 2014 16:15:38 +1000 Subject: Lambda and multi-catch In-Reply-To: References: <53185C2D.4090907@univ-mlv.fr> <531955F2.1010602@oracle.com> Message-ID: <5319640A.7060507@oracle.com> On 7/03/2014 3:53 PM, Sam Pullara wrote: > It is throwing an NPE when he dereferences the null o ? I thought the > same thing the first time I looked at it. Ughhh. That was unnecessarily obscure. So the issue is that use of Exception is catching exceptions it should not. Thanks Sam. David > Sam > > > > On Thu, Mar 6, 2014 at 9:15 PM, David Holmes > wrote: > > On 6/03/2014 9:29 PM, Remi Forax wrote: > > A colleague of mine have find a hairy bug, > > here is a simple code to reproduce it. > > > > import java.io.IOException; > > > > public class FunWithMultiCatch { > > public static void main(String[] args) { > > Runnable r = () -> { > > try { > > Object o = null; > > o.getClass(); > > throw new IOException(); > > } catch(IOException | IllegalArgumentException e) { > > System.out.println("KO !"); > > } catch(RuntimeException e) { > > System.out.println("OK !"); > > } > > }; > > r.run(); > > } > > } > > > > It prints 'KO !' :( > > What am I missing? We throw IOException and have a catch block for > IOException so I would expect it to print "KO !". > > I'm more surprised that a non-lambda version prints "OK !" > > > The problem can be reproduced for any multi-catch inside a lambda, > > the exception table is clearly wrong, the IOException and IAE are > merged > > to their common supertype (java.lang.Exception). > > That would definitely seem wrong, but more of an issue with catching > RuntimeExceptions in the first block that should be handled by the > second catch block. eg change it to throw IllegalStateException() > instead: > > if (false) throw new IOException(); > throw new IllegalStateException(); > > -> "KO !" !!!WT! > > David > ----- > > > private static void lambda$main$0(); > > Code: > > 0: aconst_null > > ... > > 14: athrow > > 15: astore_0 > > 16: getstatic #8 // Field > > java/lang/System.out:Ljava/io/PrintStream; > > 19: ldc #9 // String OK ! > > 21: invokevirtual #10 // Method > > java/io/PrintStream.println:(Ljava/lang/String;)V > > 24: goto 36 > > 27: astore_0 > > 28: getstatic #8 // Field > > java/lang/System.out:Ljava/io/PrintStream; > > 31: ldc #12 // String KO ! > > 33: invokevirtual #10 // Method > > java/io/PrintStream.println:(Ljava/lang/String;)V > > 36: return > > Exception table: > > from to target type > > 0 15 15 Class > > java/lang/Exception <------------ Oops > > 0 15 27 Class java/lang/RuntimeException > > > > The current workaround is to not use a multi-catch in a lambda. > > > > It seems that some version of Eclipse had the same issue but it's > fixed > > with the one I use (from 21 feb). > > > > cheers, > > R?mi > > > > > > From gustav.r.akesson at gmail.com Fri Mar 7 11:29:26 2014 From: gustav.r.akesson at gmail.com (=?ISO-8859-1?Q?Gustav_=C5kesson?=) Date: Fri, 7 Mar 2014 20:29:26 +0100 Subject: Lambda and multi-catch In-Reply-To: <5319640A.7060507@oracle.com> References: <53185C2D.4090907@univ-mlv.fr> <531955F2.1010602@oracle.com> <5319640A.7060507@oracle.com> Message-ID: Hi, I too was a bit confused by the example. :-) I assume this nasty bug is already registered? Best regards, Gustav ?kesson Den 7 mar 2014 07:17 skrev "David Holmes" : > On 7/03/2014 3:53 PM, Sam Pullara wrote: > > It is throwing an NPE when he dereferences the null o -- I thought the > > same thing the first time I looked at it. > > Ughhh. That was unnecessarily obscure. > > So the issue is that use of Exception is catching exceptions it should not. > > Thanks Sam. > > David > > > Sam > > > > > > > > On Thu, Mar 6, 2014 at 9:15 PM, David Holmes > > wrote: > > > > On 6/03/2014 9:29 PM, Remi Forax wrote: > > > A colleague of mine have find a hairy bug, > > > here is a simple code to reproduce it. > > > > > > import java.io.IOException; > > > > > > public class FunWithMultiCatch { > > > public static void main(String[] args) { > > > Runnable r = () -> { > > > try { > > > Object o = null; > > > o.getClass(); > > > throw new IOException(); > > > } catch(IOException | IllegalArgumentException e) { > > > System.out.println("KO !"); > > > } catch(RuntimeException e) { > > > System.out.println("OK !"); > > > } > > > }; > > > r.run(); > > > } > > > } > > > > > > It prints 'KO !' :( > > > > What am I missing? We throw IOException and have a catch block for > > IOException so I would expect it to print "KO !". > > > > I'm more surprised that a non-lambda version prints "OK !" > > > > > The problem can be reproduced for any multi-catch inside a lambda, > > > the exception table is clearly wrong, the IOException and IAE are > > merged > > > to their common supertype (java.lang.Exception). > > > > That would definitely seem wrong, but more of an issue with catching > > RuntimeExceptions in the first block that should be handled by the > > second catch block. eg change it to throw IllegalStateException() > > instead: > > > > if (false) throw new IOException(); > > throw new IllegalStateException(); > > > > -> "KO !" !!!WT! > > > > David > > ----- > > > > > private static void lambda$main$0(); > > > Code: > > > 0: aconst_null > > > ... > > > 14: athrow > > > 15: astore_0 > > > 16: getstatic #8 // Field > > > java/lang/System.out:Ljava/io/PrintStream; > > > 19: ldc #9 // String OK ! > > > 21: invokevirtual #10 // Method > > > java/io/PrintStream.println:(Ljava/lang/String;)V > > > 24: goto 36 > > > 27: astore_0 > > > 28: getstatic #8 // Field > > > java/lang/System.out:Ljava/io/PrintStream; > > > 31: ldc #12 // String KO ! > > > 33: invokevirtual #10 // Method > > > java/io/PrintStream.println:(Ljava/lang/String;)V > > > 36: return > > > Exception table: > > > from to target type > > > 0 15 15 Class > > > java/lang/Exception <------------ Oops > > > 0 15 27 Class java/lang/RuntimeException > > > > > > The current workaround is to not use a multi-catch in a lambda. > > > > > > It seems that some version of Eclipse had the same issue but it's > > fixed > > > with the one I use (from 21 feb). > > > > > > cheers, > > > R?mi > > > > > > > > > > > > From vicente.romero at oracle.com Fri Mar 7 12:34:43 2014 From: vicente.romero at oracle.com (Vicente-Arturo Romero-Zaldivar) Date: Fri, 07 Mar 2014 20:34:43 +0000 Subject: Lambda and multi-catch In-Reply-To: References: <53185C2D.4090907@univ-mlv.fr> <531955F2.1010602@oracle.com> <5319640A.7060507@oracle.com> Message-ID: <531A2D63.2040204@oracle.com> Hi Remi, Thanks for the bug report, I have filed: https://bugs.openjdk.java.net/browse/JDK-8036942 to track it, Vicente From babanin at gmail.com Fri Mar 7 23:50:38 2014 From: babanin at gmail.com (Ivan Babanin) Date: Sat, 8 Mar 2014 03:50:38 +0400 Subject: Lambda and multi-catch Message-ID: <115301cf3a60$0b50a660$21f1f320$@gmail.com> > A colleague of mine have find a hairy bug, > here is a simple code to reproduce it. > import java.io.IOException; > public class FunWithMultiCatch { > public static void main(String[] args) { > Runnable r = () -> { > try { > Object o = null; > o.getClass(); > throw new IOException(); > } catch(IOException | IllegalArgumentException e) { > System.out.println("KO !"); > } catch(RuntimeException e) { > System.out.println("OK !"); > } > }; > r.run(); > } > } > It prints 'KO !' :( > The problem can be reproduced for any multi-catch inside a lambda, > the exception table is clearly wrong, the IOException and IAE are merged > to their common supertype (java.lang.Exception). I got source from http://hg.openjdk.java.net/jdk8u/jdk8u After a brief analysis, I found the problem in LambdaToMethod. visitVarDef(...):438. Tree translator creates JCVariableDecl (line 444) and ignores vartype of original tree, that's why Gen. genCatch(...) generate only single catch in exception table. I think my code (surrounded by comments) will fix this bug: LambdaTranslationContext lambdaContext = (LambdaTranslationContext)context; if (context != null && lambdaContext.getSymbolMap(LOCAL_VAR).containsKey(tree.sym)) { JCExpression init = translate(tree.init); int prevPos = make.pos; try { result = make.at(tree).VarDef((VarSymbol)lambdaContext.getSymbolMap(LOCAL_VAR).get(tr ee.sym), init); // Copy vartype from original tree for variables with multiple types if ((result != null) && tree.vartype.hasTag(TYPEUNION)) { ((JCVariableDecl) result).vartype = tree.vartype; } // } finally { make.at(prevPos); } sy, Ivan From babanin at gmail.com Sat Mar 8 18:31:15 2014 From: babanin at gmail.com (Ivan Babanin) Date: Sat, 8 Mar 2014 22:31:15 +0400 Subject: Lambda and multi-catch Message-ID: <001a01cf3afc$980afbe0$c820f3a0$@gmail.com> > A colleague of mine have find a hairy bug, > here is a simple code to reproduce it. > import java.io.IOException; > public class FunWithMultiCatch { > ? ?public static void main(String[] args) { > ? ? ?Runnable r = () -> { > ? ? ? ?try { > ? ? ? ? ?Object o = null; > ? ? ? ? ?o.getClass(); > ? ? ? ? throw new IOException(); > ? ? ? } catch(IOException | IllegalArgumentException e) { > ? ? ? ? System.out.println("KO !"); > ? ? ? } catch(RuntimeException e) { > ? ? ? ? System.out.println("OK !"); > ? ? ? } > ? ? }; > ? ? r.run(); > ? ?} > } > It prints 'KO !' :( > The problem can be reproduced for any multi-catch inside a lambda, > the exception table is clearly wrong, the IOException and IAE are merged > to their common supertype (java.lang.Exception). I got source from?http://hg.openjdk.java.net/jdk8u/jdk8u After a brief analysis, I found the problem in LambdaToMethod.visitVarDef(...):438. Tree translator creates JCVariableDecl (line 444) and ignores vartype of original tree, that's why Gen. genCatch(...) generate only single catch in exception table. I think my code (surrounded by comments) will fix this bug: result = make.at(tree).VarDef((VarSymbol)lambdaContext.getSymbolMap(LOCAL_VAR).get(tr ee.sym), init); // Copy vartype from original tree for variables with multiple types if (tree.vartype.hasTag(TYPEUNION)) { ((JCVariableDecl) result).vartype = tree.vartype; } // sy, Ivan From vicente.romero at oracle.com Mon Mar 10 16:04:13 2014 From: vicente.romero at oracle.com (Vicente-Arturo Romero-Zaldivar) Date: Mon, 10 Mar 2014 16:04:13 +0000 Subject: Lambda and multi-catch In-Reply-To: <001a01cf3afc$980afbe0$c820f3a0$@gmail.com> References: <001a01cf3afc$980afbe0$c820f3a0$@gmail.com> Message-ID: <531DE27D.9030607@oracle.com> Hi Ivan, Thanks for your patch. After analyzing the code in LambdaToMethod, I think that this is not the best solution for the problem. We should analyze if there is a general solution here. This bug may indicate that some more statements may be affected. I will add my findings in the bug entry [1] along with your proposed patch as a reference. Thanks, Vicente [1] https://bugs.openjdk.java.net/browse/JDK-8036942 On 08/03/14 18:31, Ivan Babanin wrote: >> A colleague of mine have find a hairy bug, >> here is a simple code to reproduce it. >> import java.io.IOException; >> public class FunWithMultiCatch { >> public static void main(String[] args) { >> Runnable r = () -> { >> try { >> Object o = null; >> o.getClass(); >> throw new IOException(); >> } catch(IOException | IllegalArgumentException e) { >> System.out.println("KO !"); >> } catch(RuntimeException e) { >> System.out.println("OK !"); >> } >> }; >> r.run(); >> } >> } >> It prints 'KO !' :( >> The problem can be reproduced for any multi-catch inside a lambda, >> the exception table is clearly wrong, the IOException and IAE are merged >> to their common supertype (java.lang.Exception). > I got source from http://hg.openjdk.java.net/jdk8u/jdk8u > > After a brief analysis, I found the problem in > LambdaToMethod.visitVarDef(...):438. > Tree translator creates JCVariableDecl (line 444) and ignores vartype of > original tree, > that's why Gen. genCatch(...) generate only single catch in exception table. > > I think my code (surrounded by comments) will fix this bug: > > result = > make.at(tree).VarDef((VarSymbol)lambdaContext.getSymbolMap(LOCAL_VAR).get(tr > ee.sym), init); > // Copy vartype from original tree for variables with multiple types > if (tree.vartype.hasTag(TYPEUNION)) { > ((JCVariableDecl) result).vartype = tree.vartype; > } > // > > sy, > Ivan > > From zhong.j.yu at gmail.com Thu Mar 13 23:48:12 2014 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Thu, 13 Mar 2014 18:48:12 -0500 Subject: Overload issue - more-specific-ness Message-ID: Consider this program: // like Runnable, but throws interface RunnableX extends Callable { void run() throws Exception; default Void call() throws Exception { run(); return null; } } static void foo(RunnableX r){} static void foo(Callable> c){} The overload should be fine, because both functional interfaces are 0-arg. And foo#2 should be more specific than foo#1. However the following program fails to compile public void test() { foo( ()->new ArrayList() ); // javac: reference to foo is ambiguous // both method foo(RunnableX) and method foo(Callable>) match } Is it the correct behavior or a bug? It seems to me that foo#2 should be the most-specific method here. ( If RunnableX does NOT extend Callable, the program compiles.) Thanks, Zhong Yu From peter.levart at gmail.com Fri Mar 14 08:10:47 2014 From: peter.levart at gmail.com (Peter Levart) Date: Fri, 14 Mar 2014 09:10:47 +0100 Subject: Overload issue - more-specific-ness In-Reply-To: References: Message-ID: <5322B987.3070105@gmail.com> On 03/14/2014 12:48 AM, Zhong Yu wrote: > Consider this program: > > // like Runnable, but throws > interface RunnableX extends Callable > { > void run() throws Exception; > > default Void call() throws Exception > { > run(); > return null; > } > } > > static void foo(RunnableX r){} > static void foo(Callable> c){} > > The overload should be fine, because both functional interfaces are > 0-arg. And foo#2 should be more specific than foo#1. > > However the following program fails to compile > > public void test() > { > foo( ()->new ArrayList() ); > // javac: reference to foo is ambiguous > // both method foo(RunnableX) and method foo(Callable>) match > } > > Is it the correct behavior or a bug? It seems to me that foo#2 should > be the most-specific method here. > > ( If RunnableX does NOT extend Callable, the program compiles.) ...and selects foo(Callable). So when interfaces are not related the structural type of Callable is prefered to that of RunnableX. If RunnableX is related to (a subtype of) Callable then most specific rule should prevail. I think. But it seems that structural fitness and subtype specificness have equal weight here, so compiler is undecided. Regards, Peter > > Thanks, > Zhong Yu > From brian.goetz at oracle.com Fri Mar 14 17:48:16 2014 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 14 Mar 2014 13:48:16 -0400 Subject: Overload issue - more-specific-ness In-Reply-To: <5322B987.3070105@gmail.com> References: <5322B987.3070105@gmail.com> Message-ID: <532340E0.9090800@oracle.com> Yes, this is correct behavior. The rule is that overloading methods whose arguments are same-arity SAMs (and where other arguments can't be used to disambiguate) are going to lead to ambiguities. Bad: overloaded methods with functional interfaces of the same arity in the same position void foo(Function) void foo(ToIntFunction) Unless another non-functional argument can provide disambiguation void foo(String, Function) void foo(int, ToIntFunction) OK: overload with the same functional interface in the same position void foo(Function, String) void foo(Function, int) On 3/14/2014 4:10 AM, Peter Levart wrote: > > On 03/14/2014 12:48 AM, Zhong Yu wrote: >> Consider this program: >> >> // like Runnable, but throws >> interface RunnableX extends Callable >> { >> void run() throws Exception; >> >> default Void call() throws Exception >> { >> run(); >> return null; >> } >> } >> >> static void foo(RunnableX r){} >> static void foo(Callable> c){} >> >> The overload should be fine, because both functional interfaces are >> 0-arg. And foo#2 should be more specific than foo#1. >> >> However the following program fails to compile >> >> public void test() >> { >> foo( ()->new ArrayList() ); >> // javac: reference to foo is ambiguous >> // both method foo(RunnableX) and method >> foo(Callable>) match >> } >> >> Is it the correct behavior or a bug? It seems to me that foo#2 should >> be the most-specific method here. >> >> ( If RunnableX does NOT extend Callable, the program compiles.) > > ...and selects foo(Callable). So when interfaces are not related the > structural type of Callable is prefered to that of RunnableX. If > RunnableX is related to (a subtype of) Callable then most specific rule > should prevail. I think. But it seems that structural fitness and > subtype specificness have equal weight here, so compiler is undecided. > > Regards, Peter > >> >> Thanks, >> Zhong Yu >> > > From anna.kozlova at jetbrains.com Fri Mar 14 19:22:01 2014 From: anna.kozlova at jetbrains.com (Anna Kozlova) Date: Fri, 14 Mar 2014 20:22:01 +0100 Subject: Overload issue - more-specific-ness In-Reply-To: <532340E0.9090800@oracle.com> References: <5322B987.3070105@gmail.com> <532340E0.9090800@oracle.com> Message-ID: <098301cf3fba$ad4486d0$07cd9470$@jetbrains.com> >From spec 15.12.2.5 Choosing the Most Specific Method: In addition, a functional interface type S is more specific than a functional interface type T for an expression exp if T is not a subtype of S .... R2 is void => true Here SAM types are not subtypes from one another and one return type is void (another is not) => most specific method could be chosen. Should spec probably use |T| and |S| then? Thanks, Anna -----Original Message----- From: lambda-dev [mailto:lambda-dev-bounces at openjdk.java.net] On Behalf Of Brian Goetz Sent: Friday, March 14, 2014 6:48 PM To: Peter Levart; Zhong Yu; lambda-dev at openjdk.java.net Subject: Re: Overload issue - more-specific-ness Yes, this is correct behavior. The rule is that overloading methods whose arguments are same-arity SAMs (and where other arguments can't be used to disambiguate) are going to lead to ambiguities. Bad: overloaded methods with functional interfaces of the same arity in the same position void foo(Function) void foo(ToIntFunction) Unless another non-functional argument can provide disambiguation void foo(String, Function) void foo(int, ToIntFunction) OK: overload with the same functional interface in the same position void foo(Function, String) void foo(Function, int) On 3/14/2014 4:10 AM, Peter Levart wrote: > > On 03/14/2014 12:48 AM, Zhong Yu wrote: >> Consider this program: >> >> // like Runnable, but throws >> interface RunnableX extends Callable >> { >> void run() throws Exception; >> >> default Void call() throws Exception >> { >> run(); >> return null; >> } >> } >> >> static void foo(RunnableX r){} >> static void foo(Callable> c){} >> >> The overload should be fine, because both functional interfaces are >> 0-arg. And foo#2 should be more specific than foo#1. >> >> However the following program fails to compile >> >> public void test() >> { >> foo( ()->new ArrayList() ); >> // javac: reference to foo is ambiguous >> // both method foo(RunnableX) and method >> foo(Callable>) match >> } >> >> Is it the correct behavior or a bug? It seems to me that foo#2 should >> be the most-specific method here. >> >> ( If RunnableX does NOT extend Callable, the program compiles.) > > ...and selects foo(Callable). So when interfaces are not related the > structural type of Callable is prefered to that of RunnableX. If > RunnableX is related to (a subtype of) Callable then most specific > rule should prevail. I think. But it seems that structural fitness and > subtype specificness have equal weight here, so compiler is undecided. > > Regards, Peter > >> >> Thanks, >> Zhong Yu >> > > !DSPAM:35,5323412a213961640333909! From zhong.j.yu at gmail.com Fri Mar 14 20:41:30 2014 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Fri, 14 Mar 2014 15:41:30 -0500 Subject: Overload issue - more-specific-ness In-Reply-To: <532340E0.9090800@oracle.com> References: <5322B987.3070105@gmail.com> <532340E0.9090800@oracle.com> Message-ID: On Fri, Mar 14, 2014 at 12:48 PM, Brian Goetz wrote: > Yes, this is correct behavior. The rule is that overloading methods whose > arguments are same-arity SAMs (and where other arguments can't be used to > disambiguate) are going to lead to ambiguities. In my example the arity is 0, therefore the lambda expression is "explicit"; such overloading should be fine, according to our previous discussions. I agree that in general overloading should be avoided, it is too complicated for everybody. However I have a legitimate case here. Suppose I name the two methods differently void fooA(Runnable r){} void fooB(Callable> c){} the meaning of fooA and fooB are closely related, and it is easy for the programmer to make the mistake of invoking fooA while fooB is actually intended fooA( ()->new ArrayList() ); // compiles! Even more sinisterly, say fooA(mref) was initially passed a void-returning method reference, and later the method is refactored to return List -- fooA(mref) still passes javac silently. That is too dangerous. It's safer to overload the methods, and let javac choose the more "appropriate" method for the programmer. (The issue reported in top post does not block me from doing so, I don't have to have the subtype relationship.) Zhong Yu > > Bad: overloaded methods with functional interfaces of the same arity in the > same position > > void foo(Function) > void foo(ToIntFunction) > > Unless another non-functional argument can provide disambiguation > > void foo(String, Function) > void foo(int, ToIntFunction) > > OK: overload with the same functional interface in the same position > void foo(Function, String) > void foo(Function, int) > > > > > > > On 3/14/2014 4:10 AM, Peter Levart wrote: >> >> >> On 03/14/2014 12:48 AM, Zhong Yu wrote: >>> >>> Consider this program: >>> >>> // like Runnable, but throws >>> interface RunnableX extends Callable >>> { >>> void run() throws Exception; >>> >>> default Void call() throws Exception >>> { >>> run(); >>> return null; >>> } >>> } >>> >>> static void foo(RunnableX r){} >>> static void foo(Callable> c){} >>> >>> The overload should be fine, because both functional interfaces are >>> 0-arg. And foo#2 should be more specific than foo#1. >>> >>> However the following program fails to compile >>> >>> public void test() >>> { >>> foo( ()->new ArrayList() ); >>> // javac: reference to foo is ambiguous >>> // both method foo(RunnableX) and method >>> foo(Callable>) match >>> } >>> >>> Is it the correct behavior or a bug? It seems to me that foo#2 should >>> be the most-specific method here. >>> >>> ( If RunnableX does NOT extend Callable, the program compiles.) >> >> >> ...and selects foo(Callable). So when interfaces are not related the >> structural type of Callable is prefered to that of RunnableX. If >> RunnableX is related to (a subtype of) Callable then most specific rule >> should prevail. I think. But it seems that structural fitness and >> subtype specificness have equal weight here, so compiler is undecided. >> >> Regards, Peter >> >>> >>> Thanks, >>> Zhong Yu >>> >> >> > From zhong.j.yu at gmail.com Fri Mar 14 20:42:07 2014 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Fri, 14 Mar 2014 15:42:07 -0500 Subject: Overload issue - more-specific-ness In-Reply-To: <098301cf3fba$ad4486d0$07cd9470$@jetbrains.com> References: <5322B987.3070105@gmail.com> <532340E0.9090800@oracle.com> <098301cf3fba$ad4486d0$07cd9470$@jetbrains.com> Message-ID: On Fri, Mar 14, 2014 at 2:22 PM, Anna Kozlova wrote: > From spec 15.12.2.5 Choosing the Most Specific Method: > > In addition, a functional interface type S is more specific > than a functional interface type T for an expression exp if T is not a > subtype of S > .... > R2 is void => true That's my reading too, and I think IntelliJ got it right. > > Here SAM types are not subtypes from one another and one return type is void > (another is not) => most specific method could be chosen. Should spec > probably use |T| and |S| then? > > Thanks, > Anna > > -----Original Message----- > From: lambda-dev [mailto:lambda-dev-bounces at openjdk.java.net] On Behalf Of > Brian Goetz > Sent: Friday, March 14, 2014 6:48 PM > To: Peter Levart; Zhong Yu; lambda-dev at openjdk.java.net > Subject: Re: Overload issue - more-specific-ness > > Yes, this is correct behavior. The rule is that overloading methods whose > arguments are same-arity SAMs (and where other arguments can't be used to > disambiguate) are going to lead to ambiguities. > > Bad: overloaded methods with functional interfaces of the same arity in the > same position > > void foo(Function) > void foo(ToIntFunction) > > Unless another non-functional argument can provide disambiguation > > void foo(String, Function) > void foo(int, ToIntFunction) > > OK: overload with the same functional interface in the same position > void foo(Function, String) void foo(Function, int) > > > > > > On 3/14/2014 4:10 AM, Peter Levart wrote: >> >> On 03/14/2014 12:48 AM, Zhong Yu wrote: >>> Consider this program: >>> >>> // like Runnable, but throws >>> interface RunnableX extends Callable >>> { >>> void run() throws Exception; >>> >>> default Void call() throws Exception >>> { >>> run(); >>> return null; >>> } >>> } >>> >>> static void foo(RunnableX r){} >>> static void foo(Callable> c){} >>> >>> The overload should be fine, because both functional interfaces are >>> 0-arg. And foo#2 should be more specific than foo#1. >>> >>> However the following program fails to compile >>> >>> public void test() >>> { >>> foo( ()->new ArrayList() ); >>> // javac: reference to foo is ambiguous >>> // both method foo(RunnableX) and method >>> foo(Callable>) match >>> } >>> >>> Is it the correct behavior or a bug? It seems to me that foo#2 should >>> be the most-specific method here. >>> >>> ( If RunnableX does NOT extend Callable, the program compiles.) >> >> ...and selects foo(Callable). So when interfaces are not related the >> structural type of Callable is prefered to that of RunnableX. If >> RunnableX is related to (a subtype of) Callable then most specific >> rule should prevail. I think. But it seems that structural fitness and >> subtype specificness have equal weight here, so compiler is undecided. >> >> Regards, Peter >> >>> >>> Thanks, >>> Zhong Yu >>> >> >> > > > !DSPAM:35,5323412a213961640333909! > > From peter.levart at gmail.com Fri Mar 14 23:40:10 2014 From: peter.levart at gmail.com (Peter Levart) Date: Sat, 15 Mar 2014 00:40:10 +0100 Subject: Overload issue - more-specific-ness In-Reply-To: <098301cf3fba$ad4486d0$07cd9470$@jetbrains.com> References: <5322B987.3070105@gmail.com> <532340E0.9090800@oracle.com> <098301cf3fba$ad4486d0$07cd9470$@jetbrains.com> Message-ID: <5323935A.2060907@gmail.com> On 03/14/2014 08:22 PM, Anna Kozlova wrote: > From spec 15.12.2.5 Choosing the Most Specific Method: > > In addition, a functional interface type S is more specific > than a functional interface type T for an expression exp if T is not a > subtype of S > .... > R2 is void => true > > Here SAM types are not subtypes from one another and one return type is void > (another is not) => most specific method could be chosen. Oh, I've got it wrong. RunnableX <: Callable which is not related to Callable> !!! So according to spec: function type Callable> is more specific than function type RunnableX because RunnableX is not a subtype of Callable> and return type of RunnableX is void (and return type of Callable> is capture(List) which is not void). > Should spec > probably use |T| and |S| then? |T| meaning the raw type of T ? Assumig this interpretation and if the 2nd rule in spec stated: _In addition, a functional interface type /S/ is /more specific/ than a functional interface type /T/ for an expression /exp/ if /|T|/ is not a subtype of /|S|/ and one of the following conditions apply... _ ...then Callable> could not be more specific than RunnableX, because |RunnableX| <: |Callable>| and RunnableX could not be more specific than Callable> either because RunnableX is not a subtype of Callable> and we would get an undecided result as we currently do. So which one sounds more sound? I don't see a compelling reason to use raw types here. Regards, Peter > Thanks, > Anna > > -----Original Message----- > From: lambda-dev [mailto:lambda-dev-bounces at openjdk.java.net] On Behalf Of > Brian Goetz > Sent: Friday, March 14, 2014 6:48 PM > To: Peter Levart; Zhong Yu; lambda-dev at openjdk.java.net > Subject: Re: Overload issue - more-specific-ness > > Yes, this is correct behavior. The rule is that overloading methods whose > arguments are same-arity SAMs (and where other arguments can't be used to > disambiguate) are going to lead to ambiguities. > > Bad: overloaded methods with functional interfaces of the same arity in the > same position > > void foo(Function) > void foo(ToIntFunction) > > Unless another non-functional argument can provide disambiguation > > void foo(String, Function) > void foo(int, ToIntFunction) > > OK: overload with the same functional interface in the same position > void foo(Function, String) void foo(Function, int) > > > > > > On 3/14/2014 4:10 AM, Peter Levart wrote: >> On 03/14/2014 12:48 AM, Zhong Yu wrote: >>> Consider this program: >>> >>> // like Runnable, but throws >>> interface RunnableX extends Callable >>> { >>> void run() throws Exception; >>> >>> default Void call() throws Exception >>> { >>> run(); >>> return null; >>> } >>> } >>> >>> static void foo(RunnableX r){} >>> static void foo(Callable> c){} >>> >>> The overload should be fine, because both functional interfaces are >>> 0-arg. And foo#2 should be more specific than foo#1. >>> >>> However the following program fails to compile >>> >>> public void test() >>> { >>> foo( ()->new ArrayList() ); >>> // javac: reference to foo is ambiguous >>> // both method foo(RunnableX) and method >>> foo(Callable>) match >>> } >>> >>> Is it the correct behavior or a bug? It seems to me that foo#2 should >>> be the most-specific method here. >>> >>> ( If RunnableX does NOT extend Callable, the program compiles.) >> ...and selects foo(Callable). So when interfaces are not related the >> structural type of Callable is prefered to that of RunnableX. If >> RunnableX is related to (a subtype of) Callable then most specific >> rule should prevail. I think. But it seems that structural fitness and >> subtype specificness have equal weight here, so compiler is undecided. >> >> Regards, Peter >> >>> Thanks, >>> Zhong Yu >>> > !DSPAM:35,5323412a213961640333909! > > From forax at univ-mlv.fr Tue Mar 18 19:11:02 2014 From: forax at univ-mlv.fr (Remi Forax) Date: Tue, 18 Mar 2014 20:11:02 +0100 Subject: Java 8 is released ! Message-ID: <53289A46.4030906@univ-mlv.fr> As you all know, Java 8 is now released with lambdas as a preeminent feature. I want to personally thank every each of you that have contributed to that list taking time to read the different drafts, test the betas, report bugs, etc. We all together have created something amazing, something huge, something monumental as Mark said. I will never meet all of you in person, but please don't hesitate to come to me saying that you are one of us, I own you a beer, at least. thank you, R?mi From spullara at gmail.com Tue Mar 18 19:25:49 2014 From: spullara at gmail.com (Sam Pullara) Date: Tue, 18 Mar 2014 15:25:49 -0400 Subject: Java 8 is released ! In-Reply-To: <53289A46.4030906@univ-mlv.fr> References: <53289A46.4030906@univ-mlv.fr> Message-ID: <6DCB07B8-FC00-46C2-8A00-B145E7243F34@gmail.com> Hear, hear! ---Sent from Boxer | http://getboxer.com As you all know, Java 8 is now released with lambdas as a preeminent feature. I want to personally thank every each of you that have contributed to that list taking time to read the different drafts, test the betas, report bugs, etc. We all together have created something amazing, something huge, something monumental as Mark said. I will never meet all of you in person, but please don't hesitate to come to me saying that you are one of us, I own you a beer, at least. thank you, R?mi From martijnverburg at gmail.com Tue Mar 18 19:31:33 2014 From: martijnverburg at gmail.com (Martijn Verburg) Date: Tue, 18 Mar 2014 19:31:33 +0000 Subject: Java 8 is released ! In-Reply-To: <6DCB07B8-FC00-46C2-8A00-B145E7243F34@gmail.com> References: <53289A46.4030906@univ-mlv.fr> <6DCB07B8-FC00-46C2-8A00-B145E7243F34@gmail.com> Message-ID: +1 - deploying to our PRD environment tomorrow and there's a bunch of callback and handler code running for it's life the day after :-). Cheers, Martijn On 18 March 2014 19:25, Sam Pullara wrote: > Hear, hear! > > ---Sent from Boxer | http://getboxer.com > > As you all know, Java 8 is now released with lambdas as a preeminent > feature. > > > > I want to personally thank every each of you that have contributed to > that list taking time to read the different drafts, test the betas, > report bugs, etc. > > We all together have created something amazing, something huge, > something monumental as Mark said. > > > > I will never meet all of you in person, but please don't hesitate to > come to me saying that you are one of us, > > I own you a beer, at least. > > > > thank you, > > R?mi > > > > > > > From paulo.silveira at caelum.com.br Tue Mar 18 19:32:14 2014 From: paulo.silveira at caelum.com.br (Paulo Silveira) Date: Tue, 18 Mar 2014 16:32:14 -0300 Subject: Java 8 is released ! In-Reply-To: <53289A46.4030906@univ-mlv.fr> References: <53289A46.4030906@univ-mlv.fr> Message-ID: Congrats! It was quite incredible to be able to send emails with ideas, problems and suggetions and get pretty fast answers from the committers, including Remi and Brian. I never thought that the openjdk would be that open! -- Paulo Silveira www.caelum.com.br www.casadocodigo.com.br www.alura.com.br 2014-03-18 16:11 GMT-03:00 Remi Forax : > As you all know, Java 8 is now released with lambdas as a preeminent > feature. > > I want to personally thank every each of you that have contributed to that > list taking time to read the different drafts, test the betas, report bugs, > etc. > We all together have created something amazing, something huge, something > monumental as Mark said. > > I will never meet all of you in person, but please don't hesitate to come to > me saying that you are one of us, > I own you a beer, at least. > > thank you, > R?mi > > From howard.lovatt at gmail.com Tue Mar 18 19:34:15 2014 From: howard.lovatt at gmail.com (Howard Lovatt) Date: Wed, 19 Mar 2014 06:34:15 +1100 Subject: Java 8 is released ! In-Reply-To: <53289A46.4030906@univ-mlv.fr> References: <53289A46.4030906@univ-mlv.fr> Message-ID: <78FBA1F2-48D1-412D-849B-6C89F4175014@gmail.com> Congratulations to everyone at Oracle and on the expert group, your hard work has paid off. Sent from my iPad > On 19 Mar 2014, at 6:11 am, Remi Forax wrote: > > As you all know, Java 8 is now released with lambdas as a preeminent feature. > > I want to personally thank every each of you that have contributed to that list taking time to read the different drafts, test the betas, report bugs, etc. > We all together have created something amazing, something huge, something monumental as Mark said. > > I will never meet all of you in person, but please don't hesitate to come to me saying that you are one of us, > I own you a beer, at least. > > thank you, > R?mi > > From brian.goetz at oracle.com Tue Mar 18 20:35:47 2014 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 18 Mar 2014 16:35:47 -0400 Subject: Java 8 is released ! In-Reply-To: <53289A46.4030906@univ-mlv.fr> References: <53289A46.4030906@univ-mlv.fr> Message-ID: <5328AE23.8040503@oracle.com> Thanks Remi and others for all the kind words. This has been an exciting journey! At the risk of patting ourselves on the back, I'm very pleased with what we've accomplished here. When I started this project four years ago, the general sentiment in the community was fairly negative on the prospects for Java's future (or Oracle's ability to be a good steward for Java). While my primary goal was building the Java that I wanted to program in -- while remaining true to Java's core principles -- I must confess that my secondary goal was also making all the doubters have to eat their words. I'm pretty happy today on both counts. So many people helped make Lambda what it is; I can't possibly list them all, but I appreciate all of their contributions, from the members of the JSR-335 Expert Group [1] to my teammates at Oracle who contributed to the design and implementation [2] to the many folks in the community who participated in shaping this vision. We've accomplished something really great here. Now, let's go out and spread the word! [1] IBM (Dan Heidinga), Google (Josh Bloch, Kevin Bourrillion), JetBrains (Andrey Breslav, Anna Kozlova), Red Hat (David Lloyd), Goldman-Sachs (Vladimir Zakharov, Don Raab), Doug Lea, Bob Lee, Remi Forax, and Sam Pullara. [2] Dan Smith, Maurizio Cimadamore, Vicente Romero, Joe Darcy, Alex Buckley, Joel Franck, Robert Field, Jon Gibbons, Mike Duigou, Paul Sandoz, Henry Jen, Mark Reinhold, Guy Steele, Jean-Baptiste Tristan, John Rose, Karen Kinnear, Keith McGuigan, Sergey Kuksenko, and Aleksey Shipilev. Apologies if these lists are incomplete (as they almost surely are!) On 3/18/2014 3:11 PM, Remi Forax wrote: > As you all know, Java 8 is now released with lambdas as a preeminent > feature. > > I want to personally thank every each of you that have contributed to > that list taking time to read the different drafts, test the betas, > report bugs, etc. > We all together have created something amazing, something huge, > something monumental as Mark said. > > I will never meet all of you in person, but please don't hesitate to > come to me saying that you are one of us, > I own you a beer, at least. > > thank you, > R?mi > > From venkats at agiledeveloper.com Tue Mar 18 20:54:34 2014 From: venkats at agiledeveloper.com (Venkat Subramaniam) Date: Tue, 18 Mar 2014 21:54:34 +0100 Subject: Java 8 is released ! In-Reply-To: <5328AE23.8040503@oracle.com> References: <53289A46.4030906@univ-mlv.fr> <5328AE23.8040503@oracle.com> Message-ID: Congrats teams, a very exciting day. A big step for Java, a giant step for the developers, IMO. And I will join the list of those "doubters who eat their words." I was quite skeptic and was one of those guys picketing that this is not going to matter. Once I started digging in, I was pleasantly surprised. What drew me in was not lambdas but how it was implemented under the hood and also the lazy evaluations (Streams and all). The team deserves all the patting in the back. Congratulations again. Now go out and celebrate, take the night off :) Venkat On Mar 18, 2014, at 9:35 PM, Brian Goetz wrote: > Thanks Remi and others for all the kind words. This has been an exciting journey! > > At the risk of patting ourselves on the back, I'm very pleased with what we've accomplished here. When I started this project four years ago, the general sentiment in the community was fairly negative on the prospects for Java's future (or Oracle's ability to be a good steward for Java). While my primary goal was building the Java that I wanted to program in -- while remaining true to Java's core principles -- I must confess that my secondary goal was also making all the doubters have to eat their words. I'm pretty happy today on both counts. > > So many people helped make Lambda what it is; I can't possibly list them all, but I appreciate all of their contributions, from the members of the JSR-335 Expert Group [1] to my teammates at Oracle who contributed to the design and implementation [2] to the many folks in the community who participated in shaping this vision. > > We've accomplished something really great here. Now, let's go out and spread the word! > > > > [1] IBM (Dan Heidinga), Google (Josh Bloch, Kevin Bourrillion), JetBrains (Andrey Breslav, Anna Kozlova), Red Hat (David Lloyd), Goldman-Sachs (Vladimir Zakharov, Don Raab), Doug Lea, Bob Lee, Remi Forax, and Sam Pullara. > > [2] Dan Smith, Maurizio Cimadamore, Vicente Romero, Joe Darcy, Alex Buckley, Joel Franck, Robert Field, Jon Gibbons, Mike Duigou, Paul Sandoz, Henry Jen, Mark Reinhold, Guy Steele, Jean-Baptiste Tristan, John Rose, Karen Kinnear, Keith McGuigan, Sergey Kuksenko, and Aleksey Shipilev. > > Apologies if these lists are incomplete (as they almost surely are!) > > > On 3/18/2014 3:11 PM, Remi Forax wrote: >> As you all know, Java 8 is now released with lambdas as a preeminent >> feature. >> >> I want to personally thank every each of you that have contributed to >> that list taking time to read the different drafts, test the betas, >> report bugs, etc. >> We all together have created something amazing, something huge, >> something monumental as Mark said. >> >> I will never meet all of you in person, but please don't hesitate to >> come to me saying that you are one of us, >> I own you a beer, at least. >> >> thank you, >> R?mi >> >> > From heinz at javaspecialists.eu Tue Mar 18 21:39:14 2014 From: heinz at javaspecialists.eu (Dr Heinz M. Kabutz) Date: Tue, 18 Mar 2014 23:39:14 +0200 Subject: Java 8 is released ! In-Reply-To: <5328AE23.8040503@oracle.com> References: <53289A46.4030906@univ-mlv.fr> <5328AE23.8040503@oracle.com> Message-ID: Brilliant, thanks for your hard work. Whenever I look at code nowadays I see places where Java 8 features would have helped. Heinz On Tuesday, March 18, 2014, Brian Goetz wrote: > Thanks Remi and others for all the kind words. This has been an exciting > journey! > > At the risk of patting ourselves on the back, I'm very pleased with what > we've accomplished here. When I started this project four years ago, the > general sentiment in the community was fairly negative on the prospects for > Java's future (or Oracle's ability to be a good steward for Java). While > my primary goal was building the Java that I wanted to program in -- while > remaining true to Java's core principles -- I must confess that my > secondary goal was also making all the doubters have to eat their words. > I'm pretty happy today on both counts. > > So many people helped make Lambda what it is; I can't possibly list them > all, but I appreciate all of their contributions, from the members of the > JSR-335 Expert Group [1] to my teammates at Oracle who contributed to the > design and implementation [2] to the many folks in the community who > participated in shaping this vision. > > We've accomplished something really great here. Now, let's go out and > spread the word! > > > > [1] IBM (Dan Heidinga), Google (Josh Bloch, Kevin Bourrillion), JetBrains > (Andrey Breslav, Anna Kozlova), Red Hat (David Lloyd), Goldman-Sachs > (Vladimir Zakharov, Don Raab), Doug Lea, Bob Lee, Remi Forax, and Sam > Pullara. > > [2] Dan Smith, Maurizio Cimadamore, Vicente Romero, Joe Darcy, Alex > Buckley, Joel Franck, Robert Field, Jon Gibbons, Mike Duigou, Paul Sandoz, > Henry Jen, Mark Reinhold, Guy Steele, Jean-Baptiste Tristan, John Rose, > Karen Kinnear, Keith McGuigan, Sergey Kuksenko, and Aleksey Shipilev. > > Apologies if these lists are incomplete (as they almost surely are!) > > > On 3/18/2014 3:11 PM, Remi Forax wrote: > >> As you all know, Java 8 is now released with lambdas as a preeminent >> feature. >> >> I want to personally thank every each of you that have contributed to >> that list taking time to read the different drafts, test the betas, >> report bugs, etc. >> We all together have created something amazing, something huge, >> something monumental as Mark said. >> >> I will never meet all of you in person, but please don't hesitate to >> come to me saying that you are one of us, >> I own you a beer, at least. >> >> thank you, >> R?mi >> >> >> > -- Dr Heinz M. Kabutz (PhD CompSci) Author of "The Java(tm) Specialists' Newsletter" Sun/Oracle Java Champion 2005-2013 JavaOne Rockstar Speaker 2012 http://www.javaspecialists.eu Tel: +30 69 75 595 262 Skype: kabutz From zhong.j.yu at gmail.com Wed Mar 19 00:15:58 2014 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Tue, 18 Mar 2014 19:15:58 -0500 Subject: Java 8 is released ! In-Reply-To: References: <53289A46.4030906@univ-mlv.fr> <5328AE23.8040503@oracle.com> Message-ID: Kudos. This is a great gift to Java programmers. The community is gonna be very busy building new toys with these new tools. Zhong Yu On Tue, Mar 18, 2014 at 4:39 PM, Dr Heinz M. Kabutz wrote: > Brilliant, thanks for your hard work. Whenever I look at code nowadays I > see places where Java 8 features would have helped. > > Heinz > > On Tuesday, March 18, 2014, Brian Goetz wrote: > >> Thanks Remi and others for all the kind words. This has been an exciting >> journey! >> >> At the risk of patting ourselves on the back, I'm very pleased with what >> we've accomplished here. When I started this project four years ago, the >> general sentiment in the community was fairly negative on the prospects for >> Java's future (or Oracle's ability to be a good steward for Java). While >> my primary goal was building the Java that I wanted to program in -- while >> remaining true to Java's core principles -- I must confess that my >> secondary goal was also making all the doubters have to eat their words. >> I'm pretty happy today on both counts. >> >> So many people helped make Lambda what it is; I can't possibly list them >> all, but I appreciate all of their contributions, from the members of the >> JSR-335 Expert Group [1] to my teammates at Oracle who contributed to the >> design and implementation [2] to the many folks in the community who >> participated in shaping this vision. >> >> We've accomplished something really great here. Now, let's go out and >> spread the word! >> >> >> >> [1] IBM (Dan Heidinga), Google (Josh Bloch, Kevin Bourrillion), JetBrains >> (Andrey Breslav, Anna Kozlova), Red Hat (David Lloyd), Goldman-Sachs >> (Vladimir Zakharov, Don Raab), Doug Lea, Bob Lee, Remi Forax, and Sam >> Pullara. >> >> [2] Dan Smith, Maurizio Cimadamore, Vicente Romero, Joe Darcy, Alex >> Buckley, Joel Franck, Robert Field, Jon Gibbons, Mike Duigou, Paul Sandoz, >> Henry Jen, Mark Reinhold, Guy Steele, Jean-Baptiste Tristan, John Rose, >> Karen Kinnear, Keith McGuigan, Sergey Kuksenko, and Aleksey Shipilev. >> >> Apologies if these lists are incomplete (as they almost surely are!) >> >> >> On 3/18/2014 3:11 PM, Remi Forax wrote: >> >>> As you all know, Java 8 is now released with lambdas as a preeminent >>> feature. >>> >>> I want to personally thank every each of you that have contributed to >>> that list taking time to read the different drafts, test the betas, >>> report bugs, etc. >>> We all together have created something amazing, something huge, >>> something monumental as Mark said. >>> >>> I will never meet all of you in person, but please don't hesitate to >>> come to me saying that you are one of us, >>> I own you a beer, at least. >>> >>> thank you, >>> R?mi >>> >>> >>> >> > > -- > Dr Heinz M. Kabutz (PhD CompSci) > Author of "The Java(tm) Specialists' Newsletter" > Sun/Oracle Java Champion 2005-2013 > JavaOne Rockstar Speaker 2012 > http://www.javaspecialists.eu > Tel: +30 69 75 595 262 > Skype: kabutz > From ali.ebrahimi1781 at gmail.com Wed Mar 19 09:16:45 2014 From: ali.ebrahimi1781 at gmail.com (Ali Ebrahimi) Date: Wed, 19 Mar 2014 13:46:45 +0430 Subject: Java 8 is released ! In-Reply-To: <5328AE23.8040503@oracle.com> References: <53289A46.4030906@univ-mlv.fr> <5328AE23.8040503@oracle.com> Message-ID: Congratulations, thanks for your hard work. This is latest and best event in my latest days of old year. Friday starts Iranian's new year and you actually gave awesome new year gift for me. Thanks again and have good time. On Wed, Mar 19, 2014 at 1:05 AM, Brian Goetz wrote: > Thanks Remi and others for all the kind words. This has been an exciting > journey! > > At the risk of patting ourselves on the back, I'm very pleased with what > we've accomplished here. When I started this project four years ago, the > general sentiment in the community was fairly negative on the prospects for > Java's future (or Oracle's ability to be a good steward for Java). While > my primary goal was building the Java that I wanted to program in -- while > remaining true to Java's core principles -- I must confess that my > secondary goal was also making all the doubters have to eat their words. > I'm pretty happy today on both counts. > > So many people helped make Lambda what it is; I can't possibly list them > all, but I appreciate all of their contributions, from the members of the > JSR-335 Expert Group [1] to my teammates at Oracle who contributed to the > design and implementation [2] to the many folks in the community who > participated in shaping this vision. > > We've accomplished something really great here. Now, let's go out and > spread the word! > > > > [1] IBM (Dan Heidinga), Google (Josh Bloch, Kevin Bourrillion), JetBrains > (Andrey Breslav, Anna Kozlova), Red Hat (David Lloyd), Goldman-Sachs > (Vladimir Zakharov, Don Raab), Doug Lea, Bob Lee, Remi Forax, and Sam > Pullara. > > [2] Dan Smith, Maurizio Cimadamore, Vicente Romero, Joe Darcy, Alex > Buckley, Joel Franck, Robert Field, Jon Gibbons, Mike Duigou, Paul Sandoz, > Henry Jen, Mark Reinhold, Guy Steele, Jean-Baptiste Tristan, John Rose, > Karen Kinnear, Keith McGuigan, Sergey Kuksenko, and Aleksey Shipilev. > > Apologies if these lists are incomplete (as they almost surely are!) > > > > On 3/18/2014 3:11 PM, Remi Forax wrote: > >> As you all know, Java 8 is now released with lambdas as a preeminent >> feature. >> >> I want to personally thank every each of you that have contributed to >> that list taking time to read the different drafts, test the betas, >> report bugs, etc. >> We all together have created something amazing, something huge, >> something monumental as Mark said. >> >> I will never meet all of you in person, but please don't hesitate to >> come to me saying that you are one of us, >> I own you a beer, at least. >> >> thank you, >> R?mi >> >> >> > From jin.phd at gmail.com Wed Mar 19 09:42:37 2014 From: jin.phd at gmail.com (Jin Mingjian) Date: Wed, 19 Mar 2014 17:42:37 +0800 Subject: Java 8 is released ! In-Reply-To: References: <53289A46.4030906@univ-mlv.fr> <5328AE23.8040503@oracle.com> Message-ID: Add my congratulations here:) Do not mention "toys"! I am truly busy building new "gaint" with Java 8:) Jin On Wed, Mar 19, 2014 at 5:16 PM, Ali Ebrahimi wrote: > Congratulations, thanks for your hard work. This is latest and best event > in my latest days of old year. Friday starts Iranian's new year and you > actually gave awesome new year gift for me. Thanks again and have good > time. > > > > On Wed, Mar 19, 2014 at 1:05 AM, Brian Goetz > wrote: > > > Thanks Remi and others for all the kind words. This has been an exciting > > journey! > > > > At the risk of patting ourselves on the back, I'm very pleased with what > > we've accomplished here. When I started this project four years ago, the > > general sentiment in the community was fairly negative on the prospects > for > > Java's future (or Oracle's ability to be a good steward for Java). While > > my primary goal was building the Java that I wanted to program in -- > while > > remaining true to Java's core principles -- I must confess that my > > secondary goal was also making all the doubters have to eat their words. > > I'm pretty happy today on both counts. > > > > So many people helped make Lambda what it is; I can't possibly list them > > all, but I appreciate all of their contributions, from the members of the > > JSR-335 Expert Group [1] to my teammates at Oracle who contributed to the > > design and implementation [2] to the many folks in the community who > > participated in shaping this vision. > > > > We've accomplished something really great here. Now, let's go out and > > spread the word! > > > > > > > > [1] IBM (Dan Heidinga), Google (Josh Bloch, Kevin Bourrillion), JetBrains > > (Andrey Breslav, Anna Kozlova), Red Hat (David Lloyd), Goldman-Sachs > > (Vladimir Zakharov, Don Raab), Doug Lea, Bob Lee, Remi Forax, and Sam > > Pullara. > > > > [2] Dan Smith, Maurizio Cimadamore, Vicente Romero, Joe Darcy, Alex > > Buckley, Joel Franck, Robert Field, Jon Gibbons, Mike Duigou, Paul > Sandoz, > > Henry Jen, Mark Reinhold, Guy Steele, Jean-Baptiste Tristan, John Rose, > > Karen Kinnear, Keith McGuigan, Sergey Kuksenko, and Aleksey Shipilev. > > > > Apologies if these lists are incomplete (as they almost surely are!) > > > > > > > > On 3/18/2014 3:11 PM, Remi Forax wrote: > > > >> As you all know, Java 8 is now released with lambdas as a preeminent > >> feature. > >> > >> I want to personally thank every each of you that have contributed to > >> that list taking time to read the different drafts, test the betas, > >> report bugs, etc. > >> We all together have created something amazing, something huge, > >> something monumental as Mark said. > >> > >> I will never meet all of you in person, but please don't hesitate to > >> come to me saying that you are one of us, > >> I own you a beer, at least. > >> > >> thank you, > >> R?mi > >> > >> > >> > > > > From michael.hixson at gmail.com Wed Mar 19 10:16:38 2014 From: michael.hixson at gmail.com (Michael Hixson) Date: Wed, 19 Mar 2014 03:16:38 -0700 Subject: Java 8 is released ! In-Reply-To: References: <53289A46.4030906@univ-mlv.fr> Message-ID: Exactly! It is great to see the finished product, and the way you all managed this mailing list throughout development was fantastic. Thank you all very much for responding to our feedback. I mean, I ask a question and then people like Brian Goetz and Paul Sandoz respond with these epic *lessons* in the form of emails? How did I get so lucky? Well done, and thanks again. At work, the Java 8 migration plan is "Now." -Michael On Tue, Mar 18, 2014 at 12:32 PM, Paulo Silveira wrote: > Congrats! It was quite incredible to be able to send emails with > ideas, problems and suggetions and get pretty fast answers from the > committers, including Remi and Brian. I never thought that the openjdk > would be that open! > -- > Paulo Silveira > www.caelum.com.br > www.casadocodigo.com.br > www.alura.com.br > > > 2014-03-18 16:11 GMT-03:00 Remi Forax : >> As you all know, Java 8 is now released with lambdas as a preeminent >> feature. >> >> I want to personally thank every each of you that have contributed to that >> list taking time to read the different drafts, test the betas, report bugs, >> etc. >> We all together have created something amazing, something huge, something >> monumental as Mark said. >> >> I will never meet all of you in person, but please don't hesitate to come to >> me saying that you are one of us, >> I own you a beer, at least. >> >> thank you, >> R?mi >> >> > From marko.topolnik at gmail.com Sat Mar 22 21:31:47 2014 From: marko.topolnik at gmail.com (Marko Topolnik) Date: Sat, 22 Mar 2014 22:31:47 +0100 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: References: Message-ID: <933DB5D5-4E23-4CE0-BEDF-87FB54733651@gmail.com> I have a use case where I process a BufferedReader#lines() and each line takes a substantial amount of time (say 20 ms). The processing is easily parallelizable, however for smaller input sizes, little or no parallelization is attempted due to the batch size step of 1024 hardcoded into IteratorSpliterator when there is no size estimate. As a workaround I have coded a modified IteratorSpliterator which takes the batch size as a parameter and keeps it fixed (no arithmetic increasing). With a batch size of 100 I achieve full load on all four cores on my laptop. Since such an approach is far from elegant (taking more than 100 lines of code, which include a copy-paste of the private ArraySpliterator and the anonymous Iterator over BufferedReader's lines), I was motivated to address this mailing list in a search of a better, more idiomatic way towards achieving good parallelism for my scenario. What could I do instead of reimplementing a Spliterator from scratch? Regards, Marko Topolnik From paul.sandoz at oracle.com Mon Mar 24 09:31:31 2014 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 24 Mar 2014 10:31:31 +0100 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: <933DB5D5-4E23-4CE0-BEDF-87FB54733651@gmail.com> References: <933DB5D5-4E23-4CE0-BEDF-87FB54733651@gmail.com> Message-ID: Hi Marko, You did the right thing and wrote your own Spliterator, but why did you have to copy the ArraySpliterator code, can use the factory method on Spliterators? public static Spliterator spliterator(Object[] array, int additionalCharacteristics) The approach to extract parallelism from a sequential source was designed to not particularly favour one scenario over another (cost per element, # elements, reduction), and we did not want to expose any controls like the arithmetic progression properties since most people won't know what to do with these (in hindsight this might have been OK on the Spliterators.AbstractSpliterator implementation). Unfortunately in your case i am assuming the cost-per-line is a dominant factor over number lines to be processed. Paul. On Mar 22, 2014, at 10:31 PM, Marko Topolnik wrote: > I have a use case where I process a BufferedReader#lines() and each line takes a substantial amount of time (say 20 ms). The processing is easily parallelizable, however for smaller input sizes, little or no parallelization is attempted due to the batch size step of 1024 hardcoded into IteratorSpliterator when there is no size estimate. > > As a workaround I have coded a modified IteratorSpliterator which takes the batch size as a parameter and keeps it fixed (no arithmetic increasing). With a batch size of 100 I achieve full load on all four cores on my laptop. > > Since such an approach is far from elegant (taking more than 100 lines of code, which include a copy-paste of the private ArraySpliterator and the anonymous Iterator over BufferedReader's lines), I was motivated to address this mailing list in a search of a better, more idiomatic way towards achieving good parallelism for my scenario. What could I do instead of reimplementing a Spliterator from scratch? > > Regards, > Marko Topolnik > > From marko.topolnik at gmail.com Mon Mar 24 14:12:26 2014 From: marko.topolnik at gmail.com (Marko Topolnik) Date: Mon, 24 Mar 2014 15:12:26 +0100 Subject: Exploiting concurrency with IteratorSpliterator of unknown size Message-ID: Hi Paul, > You did the right thing and wrote your own Spliterator, but why did you have > to copy the ArraySpliterator code, can use the factory method on Spliterators? Yes, I realized this soon after posting here. I was initially following the original IteratorSpliterator implementation, which makes a direct call to the constructor. > The approach to extract parallelism from a sequential source was designed to > not particularly favour one scenario over another (cost per element, # elements, > reduction), and we did not want to expose any controls like the arithmetic > progression properties since most people won't know what to do with these > (in hindsight this might have been OK on the Spliterators.AbstractSpliterator > implementation). What I would generally like to find in the library is some sort of a template (abstract class) to make a custom spliterator policy easier to implement. I realize it is not trivial to get such an API right, making it serve as many needs as possible without an explosion of complexity. In the specific example of BufferedReader.lines there is the additional issue of the anonymous Iterator implementation which is burried in the lib. On a more general level, the problem may be with the way a Spliterator couples the concerns of splitting and iteration. A Spliterator based on the classic Iterator solves a part of this problem because iterating is delegated to a separate object; however the Iterator interface itself is quite cumbersome because it forces us to prefetch the next item and cache it when hasNext is called. So if there was a partial Spliterator interface (which Spliterator itself would then extend) which had only tryAdvance and forEachRemaining, and if an implementation of that could be directly obtained from BufferedReader and other similar objects, then perhaps providing our own Spliterator implementation in terms of that would be nicer. > Unfortunately in your case i am assuming the cost-per-line is a dominant > factor over number lines to be processed. Yes, that is what's happening. There is one other thing I'm wondering about: couldn't the ArraySpliterator returned from IteratorSpliterator be further split into smaller chunks? Generally, how do the "recursive decomposition" mechanics play out in our scenario here? Regards, Marko Topolnik From paul.sandoz at oracle.com Mon Mar 24 15:20:00 2014 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 24 Mar 2014 16:20:00 +0100 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: References: Message-ID: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> On Mar 24, 2014, at 3:12 PM, Marko Topolnik wrote: >> The approach to extract parallelism from a sequential source was designed > to >> not particularly favour one scenario over another (cost per element, # > elements, >> reduction), and we did not want to expose any controls like the > arithmetic >> progression properties since most people won't know what to do with these >> (in hindsight this might have been OK on the > Spliterators.AbstractSpliterator >> implementation). > > What I would generally like to find in the library is some sort of a > template > (abstract class) to make a custom spliterator policy easier to implement. I > realize it is not trivial to get such an API right, making it serve as many > needs > as possible without an explosion of complexity. > That is exactly what Spliterator is for; we cannot know all splitting policies and provide partial implementations. > In the specific example of BufferedReader.lines there is the additional > issue of > the anonymous Iterator implementation which is burried in the lib. > Note that we could (and probably should) also implement that Spliterator directly leveraging Spliterators.AbstractSpliterator to reduce the layering, but still if you look at the code you will notice some internal layering with respect to managing results from calling tryAdvance when splitting. > On a more general level, the problem may be with the way a Spliterator > couples > the concerns of splitting and iteration. A Spliterator based on the classic > Iterator > solves a part of this problem because iterating is delegated to a separate > object; > however the Iterator interface itself is quite cumbersome because it forces > us to > prefetch the next item and cache it when hasNext is called. > Yes, that is one reason why we did not base Spliterator on Iterator. We did not have a compelling use-case to separate out the iteration from splitting. > So if there was a partial Spliterator interface (which Spliterator itself > would then > extend) which had only tryAdvance and forEachRemaining, I get what you are saying but that is just a Spliterator that does not split :-) > and if an > implementation of that could be directly obtained from BufferedReader and > other similar objects, > then perhaps providing our own Spliterator implementation in terms of that > would be > nicer. > >> Unfortunately in your case i am assuming the cost-per-line is a dominant >> factor over number lines to be processed. > > Yes, that is what's happening. There is one other thing I'm wondering about: > couldn't the ArraySpliterator returned from IteratorSpliterator be further > split into > smaller chunks? Yes, possibly (see below for why not for streams). > Generally, how do the "recursive decomposition" mechanics > play > out in our scenario here? > We are getting into gritty details of the implementation... If a size estimate of the root spliterator is known then a threshold is calculated so that a spliterator is no longer split if it's size estimate is below that threshold. For something like an ArrayList this will create a balanced tree where the leaf nodes are in proportion to the targeted parallelism level of the fork/join common pool. In your situation it is a worst case scenario :-) a sequential source of unknown size, This will create a right balanced tree, each left leaf node is a prefix of elements copied into an array. Because the size is unknown the Spliterator wrapping that array will not be split. Splitting of the right tree nodes will occur until a null is returned (i.e. when there are no more elements). Reduction operations can perform poorly on such trees due to the imbalance. Generally the technique of splitting from a sequential source is to try and ramp up quickly extracting out some work to warm things up while in parallel further splitting (safely) from the source. As you have found out YMMV. Paul. From brian.goetz at oracle.com Mon Mar 24 16:47:28 2014 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 24 Mar 2014 09:47:28 -0700 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> References: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> Message-ID: <2B0CB845-5456-4C34-964A-34FF7C6CF7A6@oracle.com> ?More control over splitting heuristics? has been a repeated request. Our split heuristics are tuned for low-latency, low-Q. Folks who are looking for IO parallelism rather than data parallelism, or have low-N high-Q problems, will find them ill-tuned. The obvious question is ?why not just make it a pluggable API, then?? In a perfect world, we would have. But we felt it was more important to get a clean, simple API out the door so people had *something*, and delaying 8 further for this would have been a poor choice. Its on our list to explore what we can do to improve things for problems that don?t fit into the sweet spot of the current approach. On Mar 24, 2014, at 8:20 AM, Paul Sandoz wrote: > On Mar 24, 2014, at 3:12 PM, Marko Topolnik wrote: >>> The approach to extract parallelism from a sequential source was designed >> to >>> not particularly favour one scenario over another (cost per element, # >> elements, >>> reduction), and we did not want to expose any controls like the >> arithmetic >>> progression properties since most people won't know what to do with these >>> (in hindsight this might have been OK on the >> Spliterators.AbstractSpliterator >>> implementation). >> >> What I would generally like to find in the library is some sort of a >> template >> (abstract class) to make a custom spliterator policy easier to implement. I >> realize it is not trivial to get such an API right, making it serve as many >> needs >> as possible without an explosion of complexity. >> > > That is exactly what Spliterator is for; we cannot know all splitting policies and provide partial implementations. > > >> In the specific example of BufferedReader.lines there is the additional >> issue of >> the anonymous Iterator implementation which is burried in the lib. >> > > Note that we could (and probably should) also implement that Spliterator directly leveraging Spliterators.AbstractSpliterator to reduce the layering, but still if you look at the code you will notice some internal layering with respect to managing results from calling tryAdvance when splitting. > > >> On a more general level, the problem may be with the way a Spliterator >> couples >> the concerns of splitting and iteration. A Spliterator based on the classic >> Iterator >> solves a part of this problem because iterating is delegated to a separate >> object; >> however the Iterator interface itself is quite cumbersome because it forces >> us to >> prefetch the next item and cache it when hasNext is called. >> > > Yes, that is one reason why we did not base Spliterator on Iterator. We did not have a compelling use-case to separate out the iteration from splitting. > > >> So if there was a partial Spliterator interface (which Spliterator itself >> would then >> extend) which had only tryAdvance and forEachRemaining, > > I get what you are saying but that is just a Spliterator that does not split :-) > > >> and if an >> implementation of that could be directly obtained from BufferedReader and >> other similar objects, >> then perhaps providing our own Spliterator implementation in terms of that >> would be >> nicer. >> > >>> Unfortunately in your case i am assuming the cost-per-line is a dominant >>> factor over number lines to be processed. >> >> Yes, that is what's happening. There is one other thing I'm wondering about: >> couldn't the ArraySpliterator returned from IteratorSpliterator be further >> split into >> smaller chunks? > > Yes, possibly (see below for why not for streams). > > >> Generally, how do the "recursive decomposition" mechanics >> play >> out in our scenario here? >> > > We are getting into gritty details of the implementation... > > If a size estimate of the root spliterator is known then a threshold is calculated so that a spliterator is no longer split if it's size estimate is below that threshold. For something like an ArrayList this will create a balanced tree where the leaf nodes are in proportion to the targeted parallelism level of the fork/join common pool. > > In your situation it is a worst case scenario :-) a sequential source of unknown size, This will create a right balanced tree, each left leaf node is a prefix of elements copied into an array. Because the size is unknown the Spliterator wrapping that array will not be split. Splitting of the right tree nodes will occur until a null is returned (i.e. when there are no more elements). Reduction operations can perform poorly on such trees due to the imbalance. > > Generally the technique of splitting from a sequential source is to try and ramp up quickly extracting out some work to warm things up while in parallel further splitting (safely) from the source. As you have found out YMMV. > > Paul. > From brian.goetz at oracle.com Mon Mar 24 16:50:33 2014 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 24 Mar 2014 09:50:33 -0700 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> References: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> Message-ID: ?More control over splitting heuristics? has been a repeated request. Our split heuristics are tuned for low-latency, low-Q. Folks who are looking for IO parallelism rather than data parallelism, or have low-N high-Q problems, will find them ill-tuned. The obvious question is ?why not just make it a pluggable API, then?? In a perfect world, we would have. But we felt it was more important to get a clean, simple API out the door so people had *something*, and delaying 8 further for this would have been a poor choice. Its on our list to explore what we can do to improve things for problems that don?t fit into the sweet spot of the current approach. On Mar 24, 2014, at 8:20 AM, Paul Sandoz wrote: > On Mar 24, 2014, at 3:12 PM, Marko Topolnik wrote: >>> The approach to extract parallelism from a sequential source was designed >> to >>> not particularly favour one scenario over another (cost per element, # >> elements, >>> reduction), and we did not want to expose any controls like the >> arithmetic >>> progression properties since most people won't know what to do with these >>> (in hindsight this might have been OK on the >> Spliterators.AbstractSpliterator >>> implementation). >> >> What I would generally like to find in the library is some sort of a >> template >> (abstract class) to make a custom spliterator policy easier to implement. I >> realize it is not trivial to get such an API right, making it serve as many >> needs >> as possible without an explosion of complexity. >> > > That is exactly what Spliterator is for; we cannot know all splitting policies and provide partial implementations. > > >> In the specific example of BufferedReader.lines there is the additional >> issue of >> the anonymous Iterator implementation which is burried in the lib. >> > > Note that we could (and probably should) also implement that Spliterator directly leveraging Spliterators.AbstractSpliterator to reduce the layering, but still if you look at the code you will notice some internal layering with respect to managing results from calling tryAdvance when splitting. > > >> On a more general level, the problem may be with the way a Spliterator >> couples >> the concerns of splitting and iteration. A Spliterator based on the classic >> Iterator >> solves a part of this problem because iterating is delegated to a separate >> object; >> however the Iterator interface itself is quite cumbersome because it forces >> us to >> prefetch the next item and cache it when hasNext is called. >> > > Yes, that is one reason why we did not base Spliterator on Iterator. We did not have a compelling use-case to separate out the iteration from splitting. > > >> So if there was a partial Spliterator interface (which Spliterator itself >> would then >> extend) which had only tryAdvance and forEachRemaining, > > I get what you are saying but that is just a Spliterator that does not split :-) > > >> and if an >> implementation of that could be directly obtained from BufferedReader and >> other similar objects, >> then perhaps providing our own Spliterator implementation in terms of that >> would be >> nicer. >> > >>> Unfortunately in your case i am assuming the cost-per-line is a dominant >>> factor over number lines to be processed. >> >> Yes, that is what's happening. There is one other thing I'm wondering about: >> couldn't the ArraySpliterator returned from IteratorSpliterator be further >> split into >> smaller chunks? > > Yes, possibly (see below for why not for streams). > > >> Generally, how do the "recursive decomposition" mechanics >> play >> out in our scenario here? >> > > We are getting into gritty details of the implementation... > > If a size estimate of the root spliterator is known then a threshold is calculated so that a spliterator is no longer split if it's size estimate is below that threshold. For something like an ArrayList this will create a balanced tree where the leaf nodes are in proportion to the targeted parallelism level of the fork/join common pool. > > In your situation it is a worst case scenario :-) a sequential source of unknown size, This will create a right balanced tree, each left leaf node is a prefix of elements copied into an array. Because the size is unknown the Spliterator wrapping that array will not be split. Splitting of the right tree nodes will occur until a null is returned (i.e. when there are no more elements). Reduction operations can perform poorly on such trees due to the imbalance. > > Generally the technique of splitting from a sequential source is to try and ramp up quickly extracting out some work to warm things up while in parallel further splitting (safely) from the source. As you have found out YMMV. > > Paul. > From marko.topolnik at gmail.com Mon Mar 24 20:58:05 2014 From: marko.topolnik at gmail.com (Marko Topolnik) Date: Mon, 24 Mar 2014 21:58:05 +0100 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> References: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> Message-ID: > Yes, that is one reason why we did not base Spliterator on Iterator. We did >not have a compelling use-case to separate out the iteration from splitting. > ... > I get what you are saying but that is just a Spliterator that does not split :-) Seems that here we've talked past each other... my message was that, for the same reason that the IteratorSpliterator is useful now, a partial Spliterator interface (itself _not_ a Spliterator) which handles just the iteration aspect would be useful as a building block for the full Spliterator, and better than an Iterator due to the saner API (hasNext+next in one method, efficient iteration over all elements). So if all the relevant objects, such as BufferedReader in our case, provided that partial implementation in addition to the finished, defaults-based Stream, it would be a good basis for a custom Spliterator. > > Generally, how do the "recursive decomposition" mechanics > > play > > out in our scenario here? > > > > We are getting into gritty details of the implementation... > > If a size estimate of the root spliterator is known then a threshold is >calculated so that a spliterator is no longer split if it's size estimate is below >that threshold. For something like an ArrayList this will create a balanced >tree where the leaf nodes are in proportion > to the targeted parallelism level of the fork/join common pool. > > In your situation it is a worst case scenario :-) a sequential source of >unknown size, This will create a right balanced tree, each left leaf node is a >prefix of elements copied into an array. Because the size is unknown the >Spliterator wrapping that array will not be split. Splitting of the right tree >nodes will occur until a null is returned (i.e. when there are no more >elements). Reduction operations can perform poorly on such trees due to > the imbalance. > > Generally the technique of splitting from a sequential source is to try and >ramp up quickly extracting out some work to warm things up while in parallel >further splitting (safely) from the source. As you have found out YMMV. I see; the premise behind this is that extracting an item from the sequential source has the same order of cost as its processing. As soon as processing cost ovewhelms reading cost, ramp-up loses significance because the threadpool's queue can be quickly built up by reading into ArraySpliterators. Therefore the story applies only to the contact area between two ranges of problems: those with I/O as the bottleneck and those with CPU as the bottleneck. The former is clearly not of interest here, but in the latter I see a different story explaining the effectiveness of the ramp-up strategy: it strikes a balance between the needs of high-Q and low-Q problems. For both classes there is a lower bound for N at which this strategy shows satisfactory performance, that N being a function of the batch size increment. With my problem, I unfortunately find myself below that threshold. I still seem to be missing an important piece of the puzzle in my understanding of F/J, so perhaps you can clear this up for me: it currently seems to me that, once the processing of a batch has commenced, after say 10-20 ms a sort of self-inspection could be performed by considering the count of elements processed so far. If there is still a lot more to do, further tasks could be split off and enqueued so that other threads may steal them. The size of the splits could be informed by the number of elements that were processed. I would like to know whether this is a) impossible, b) possible but would be ineffective, c) possible and potentially effective, but not yet considered for implementation, d) something else entirely? I remember reading a similar idea in this or the EG mailing list, but it was left sort of hanging in the air. On Mon, Mar 24, 2014 at 4:20 PM, Paul Sandoz wrote: > On Mar 24, 2014, at 3:12 PM, Marko Topolnik wrote: >>> The approach to extract parallelism from a sequential source was designed >> to >>> not particularly favour one scenario over another (cost per element, # >> elements, >>> reduction), and we did not want to expose any controls like the >> arithmetic >>> progression properties since most people won't know what to do with these >>> (in hindsight this might have been OK on the >> Spliterators.AbstractSpliterator >>> implementation). >> >> What I would generally like to find in the library is some sort of a >> template >> (abstract class) to make a custom spliterator policy easier to implement. I >> realize it is not trivial to get such an API right, making it serve as many >> needs >> as possible without an explosion of complexity. >> > > That is exactly what Spliterator is for; we cannot know all splitting policies and provide partial implementations. > > >> In the specific example of BufferedReader.lines there is the additional >> issue of >> the anonymous Iterator implementation which is burried in the lib. >> > > Note that we could (and probably should) also implement that Spliterator directly leveraging Spliterators.AbstractSpliterator to reduce the layering, but still if you look at the code you will notice some internal layering with respect to managing results from calling tryAdvance when splitting. > > >> On a more general level, the problem may be with the way a Spliterator >> couples >> the concerns of splitting and iteration. A Spliterator based on the classic >> Iterator >> solves a part of this problem because iterating is delegated to a separate >> object; >> however the Iterator interface itself is quite cumbersome because it forces >> us to >> prefetch the next item and cache it when hasNext is called. >> > > Yes, that is one reason why we did not base Spliterator on Iterator. We did not have a compelling use-case to separate out the iteration from splitting. > > >> So if there was a partial Spliterator interface (which Spliterator itself >> would then >> extend) which had only tryAdvance and forEachRemaining, > > I get what you are saying but that is just a Spliterator that does not split :-) > > >> and if an >> implementation of that could be directly obtained from BufferedReader and >> other similar objects, >> then perhaps providing our own Spliterator implementation in terms of that >> would be >> nicer. >> > >>> Unfortunately in your case i am assuming the cost-per-line is a dominant >>> factor over number lines to be processed. >> >> Yes, that is what's happening. There is one other thing I'm wondering about: >> couldn't the ArraySpliterator returned from IteratorSpliterator be further >> split into >> smaller chunks? > > Yes, possibly (see below for why not for streams). > > >> Generally, how do the "recursive decomposition" mechanics >> play >> out in our scenario here? >> > > We are getting into gritty details of the implementation... > > If a size estimate of the root spliterator is known then a threshold is calculated so that a spliterator is no longer split if it's size estimate is below that threshold. For something like an ArrayList this will create a balanced tree where the leaf nodes are in proportion to the targeted parallelism level of the fork/join common pool. > > In your situation it is a worst case scenario :-) a sequential source of unknown size, This will create a right balanced tree, each left leaf node is a prefix of elements copied into an array. Because the size is unknown the Spliterator wrapping that array will not be split. Splitting of the right tree nodes will occur until a null is returned (i.e. when there are no more elements). Reduction operations can perform poorly on such trees due to the imbalance. > > Generally the technique of splitting from a sequential source is to try and ramp up quickly extracting out some work to warm things up while in parallel further splitting (safely) from the source. As you have found out YMMV. > > Paul. > From marko.topolnik at gmail.com Mon Mar 24 20:59:30 2014 From: marko.topolnik at gmail.com (Marko Topolnik) Date: Mon, 24 Mar 2014 21:59:30 +0100 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: References: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> Message-ID: On Mon, Mar 24, 2014 at 5:50 PM, Brian Goetz wrote: > ?More control over splitting heuristics? has been a repeated request. Our >split heuristics are tuned for low-latency, low-Q. Folks who are looking for >IO parallelism rather than data parallelism, or have low-N high-Q problems, >will find them ill-tuned. Is this backed by a finding that most problems fall into that class? Another reason I could see applying here is that you wanted to make positively sure that no use case experiences a slow-down compared to sequential processing, and the one in danger of that is precisely the low-Q class. On the other hand, it would seem to me that at least a large proportion of map-reduce problems imply significant processing per element. > The obvious question is ?why not just make it a pluggable API, then?? In a >perfect world, we would have. But we felt it was more important to get a >clean, simple API out the door so people had *something*, and delaying 8 >further for this would have been a poor choice. Rest assured that I fully appreciate and respect the magnitude of your efforts, as well as your uniquely sensitive position. The cost of any mistake you make in the APIs is the highest possible: it will stick forever and the largest possible number of programmers will be exposed to it. > Its on our list to explore what we can do to improve things for problems that don?t fit into the sweet spot of the current approach. I still have hope that good heuristics could cover quite a lot of ground without pluaggability/parametrization. On Mon, Mar 24, 2014 at 5:50 PM, Brian Goetz wrote: > ?More control over splitting heuristics? has been a repeated request. Our split heuristics are tuned for low-latency, low-Q. Folks who are looking for IO parallelism rather than data parallelism, or have low-N high-Q problems, will find them ill-tuned. > > The obvious question is ?why not just make it a pluggable API, then?? In a perfect world, we would have. But we felt it was more important to get a clean, simple API out the door so people had *something*, and delaying 8 further for this would have been a poor choice. > > Its on our list to explore what we can do to improve things for problems that don?t fit into the sweet spot of the current approach. > > On Mar 24, 2014, at 8:20 AM, Paul Sandoz wrote: > >> On Mar 24, 2014, at 3:12 PM, Marko Topolnik wrote: >>>> The approach to extract parallelism from a sequential source was designed >>> to >>>> not particularly favour one scenario over another (cost per element, # >>> elements, >>>> reduction), and we did not want to expose any controls like the >>> arithmetic >>>> progression properties since most people won't know what to do with these >>>> (in hindsight this might have been OK on the >>> Spliterators.AbstractSpliterator >>>> implementation). >>> >>> What I would generally like to find in the library is some sort of a >>> template >>> (abstract class) to make a custom spliterator policy easier to implement. I >>> realize it is not trivial to get such an API right, making it serve as many >>> needs >>> as possible without an explosion of complexity. >>> >> >> That is exactly what Spliterator is for; we cannot know all splitting policies and provide partial implementations. >> >> >>> In the specific example of BufferedReader.lines there is the additional >>> issue of >>> the anonymous Iterator implementation which is burried in the lib. >>> >> >> Note that we could (and probably should) also implement that Spliterator directly leveraging Spliterators.AbstractSpliterator to reduce the layering, but still if you look at the code you will notice some internal layering with respect to managing results from calling tryAdvance when splitting. >> >> >>> On a more general level, the problem may be with the way a Spliterator >>> couples >>> the concerns of splitting and iteration. A Spliterator based on the classic >>> Iterator >>> solves a part of this problem because iterating is delegated to a separate >>> object; >>> however the Iterator interface itself is quite cumbersome because it forces >>> us to >>> prefetch the next item and cache it when hasNext is called. >>> >> >> Yes, that is one reason why we did not base Spliterator on Iterator. We did not have a compelling use-case to separate out the iteration from splitting. >> >> >>> So if there was a partial Spliterator interface (which Spliterator itself >>> would then >>> extend) which had only tryAdvance and forEachRemaining, >> >> I get what you are saying but that is just a Spliterator that does not split :-) >> >> >>> and if an >>> implementation of that could be directly obtained from BufferedReader and >>> other similar objects, >>> then perhaps providing our own Spliterator implementation in terms of that >>> would be >>> nicer. >>> >> >>>> Unfortunately in your case i am assuming the cost-per-line is a dominant >>>> factor over number lines to be processed. >>> >>> Yes, that is what's happening. There is one other thing I'm wondering about: >>> couldn't the ArraySpliterator returned from IteratorSpliterator be further >>> split into >>> smaller chunks? >> >> Yes, possibly (see below for why not for streams). >> >> >>> Generally, how do the "recursive decomposition" mechanics >>> play >>> out in our scenario here? >>> >> >> We are getting into gritty details of the implementation... >> >> If a size estimate of the root spliterator is known then a threshold is calculated so that a spliterator is no longer split if it's size estimate is below that threshold. For something like an ArrayList this will create a balanced tree where the leaf nodes are in proportion to the targeted parallelism level of the fork/join common pool. >> >> In your situation it is a worst case scenario :-) a sequential source of unknown size, This will create a right balanced tree, each left leaf node is a prefix of elements copied into an array. Because the size is unknown the Spliterator wrapping that array will not be split. Splitting of the right tree nodes will occur until a null is returned (i.e. when there are no more elements). Reduction operations can perform poorly on such trees due to the imbalance. >> >> Generally the technique of splitting from a sequential source is to try and ramp up quickly extracting out some work to warm things up while in parallel further splitting (safely) from the source. As you have found out YMMV. >> >> Paul. >> > > From paul.sandoz at oracle.com Tue Mar 25 10:06:31 2014 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 25 Mar 2014 11:06:31 +0100 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: References: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> Message-ID: On Mar 24, 2014, at 9:58 PM, Marko Topolnik wrote: >> Yes, that is one reason why we did not base Spliterator on Iterator. We did >> not have a compelling use-case to separate out the iteration from splitting. >> ... >> I get what you are saying but that is just a Spliterator that does not split :-) > > Seems that here we've talked past each other... my message was that, > for the same reason that the IteratorSpliterator is useful now, a > partial Spliterator interface (itself _not_ a Spliterator) which > handles just the iteration aspect would be useful as a building block > for the full Spliterator, and better than an Iterator due to the saner > API (hasNext+next in one method, efficient iteration over all > elements). So if all the relevant objects, such as BufferedReader in > our case, provided that partial implementation in addition to the > finished, defaults-based Stream, it would be a good basis for a custom > Spliterator. > I am saying we have all the building blocks to do that today, but perhaps not quite as clean as you indicate. It's easy to implement a spliterator whose trySplit returns null, and an instance of that spliterator could be wrapped in another that extracts the parallelism. It is also easy to extend from Spliterator.AbstractSpliterator: public Stream lines() { Spliterator sp = new Spliterators.AbstractSpliterator( Long.MAX_VALUE, Spliterator.ORDERED | Spliterator.NONNULL) { @Override public boolean tryAdvance(Consumer action) { try { String line = readLine(); if (line != null) { action.accept(line); return true; } else { return false; } } catch (IOException e) { throw new UncheckedIOException(e); } } @Override public void forEachRemaining(Consumer action) { try { String line; while ((line = readLine()) != null) { action.accept(line); } } catch (IOException e) { throw new UncheckedIOException(e); } } }; return StreamSupport.stream(sp, false); } I suspect we did not do that at the time because Spliterators.AbstractSpliterator may not have been in the API when BufferedReader.lines() was written. >>> Generally, how do the "recursive decomposition" mechanics >>> play >>> out in our scenario here? >>> >> >> We are getting into gritty details of the implementation... >> >> If a size estimate of the root spliterator is known then a threshold is >> calculated so that a spliterator is no longer split if it's size estimate is below >> that threshold. For something like an ArrayList this will create a balanced >> tree where the leaf nodes are in proportion >> to the targeted parallelism level of the fork/join common pool. >> >> In your situation it is a worst case scenario :-) a sequential source of >> unknown size, This will create a right balanced tree, each left leaf node is a >> prefix of elements copied into an array. Because the size is unknown the >> Spliterator wrapping that array will not be split. Splitting of the right tree >> nodes will occur until a null is returned (i.e. when there are no more >> elements). Reduction operations can perform poorly on such trees due to >> the imbalance. >> >> Generally the technique of splitting from a sequential source is to try and >> ramp up quickly extracting out some work to warm things up while in parallel >> further splitting (safely) from the source. As you have found out YMMV. > > I see; the premise behind this is that extracting an item from the > sequential source has the same order of cost as its processing. As > soon as processing cost ovewhelms reading cost, ramp-up loses > significance because the threadpool's queue can be quickly built up by > reading into ArraySpliterators. Therefore the story applies only to > the contact area between two ranges of problems: those with I/O as the > bottleneck and those with CPU as the bottleneck. The former is clearly > not of interest here, but in the latter I see a different story > explaining the effectiveness of the ramp-up strategy: it strikes a > balance between the needs of high-Q and low-Q problems. For both > classes there is a lower bound for N at which this strategy shows > satisfactory performance, that N being a function of the batch size > increment. With my problem, I unfortunately find myself below that > threshold. Yes, unfortunately so. Originally we used an arithmetic progression with a difference of 1, but this could create unduly large trees resulting in performance issues when reduced/collected. The original stream F/J code (see AbstractTask) was too aggressive splitting for such cases and Doug solved this by a rather clever but simple solution of alternating forking and computation of the left and right tasks, which meant the initiating invocation thread was no longer aggressively spawning left nodes as fast as it could (and potentially using up lots of memory). There is definitely room for improvement here. > > I still seem to be missing an important piece of the puzzle in my > understanding of F/J, so perhaps you can clear this up for me: it > currently seems to me that, once the processing of a batch has > commenced, after say 10-20 ms a sort of self-inspection could be > performed by considering the count of elements processed so far. If > there is still a lot more to do, further tasks could be split off and > enqueued so that other threads may steal them. The size of the splits > could be informed by the number of elements that were processed. I > would like to know whether this is a) impossible, b) possible but > would be ineffective, c) possible and potentially effective, but not > yet considered for implementation, d) something else entirely? I doubt it is impossible, but I don't know how effective it might be until one gets the results of such an experiment. We don't need this kind of thing for an ArrayList, and we might not need it when the size or size estimate of the sequential source is known, if N is "small" we could bias towards high-Q. We are dealing with a worst-case scenario of a sequentially ordered source of unknown size. Some simple holder of a count e.g. LongAdder or AtomicLong could be updated when each leaf node completes, plus there might be stats from the F/J pool that could be used. We don't even need to have an accurate count, perhaps an estimate (size estimate from the spliterator) is sufficient. An estimate is cheaper since we can more easily leverage forEachRemaining without wrapping with a local counting consumer. A tricky aspect would be ensuring this statistical approach is bound with the particular style of source as we don't want to hard code this in. There is also a tricker more general problem: that of automatically working out when to go parallel or sequential. I think that requires gathering statistics over multiple executions of the pipeline from which it is decided to switch modes on subsequent executions (and more generally this could also be applied to optimizing the pipeline, compiling to byte code and/or executing on say a GPU). > I > remember reading a similar idea in this or the EG mailing list, but it > was left sort of hanging in the air. > Quite possibly, i don't recall such a discussion :-) Paul. From forum at x9c.fr Tue Mar 25 10:10:40 2014 From: forum at x9c.fr (forum at x9c.fr) Date: Tue, 25 Mar 2014 10:10:40 +0000 Subject: Class file format specification for Java 8? Message-ID: <71108AD9-B053-44C2-9230-AEDBA187CB91@x9c.fr> Dear List, Now that Java 8 is released, I may want to update some of my tools to properly support this version. To this end, I wonder if there is an updated version of the "class file format specification", or any kind of delta against the version for 1.7. Thanks in advance for any pointer. Kind regards, Xavier Clerc From paul.sandoz at oracle.com Tue Mar 25 10:35:52 2014 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 25 Mar 2014 11:35:52 +0100 Subject: Class file format specification for Java 8? In-Reply-To: <71108AD9-B053-44C2-9230-AEDBA187CB91@x9c.fr> References: <71108AD9-B053-44C2-9230-AEDBA187CB91@x9c.fr> Message-ID: On Mar 25, 2014, at 11:10 AM, forum at x9c.fr wrote: > Dear List, > > Now that Java 8 is released, I may want to update some of my > tools to properly support this version. To this end, I wonder if > there is an updated version of the "class file format specification", > or any kind of delta against the version for 1.7. > > Thanks in advance for any pointer. > From here: http://docs.oracle.com/javase/specs/ you can get a diff of the changes to JVM spec from 7 to 8: http://docs.oracle.com/javase/specs/jvms/se8/jvms8-diffs.pdf Paul. From forum at x9c.fr Tue Mar 25 11:44:54 2014 From: forum at x9c.fr (forum at x9c.fr) Date: Tue, 25 Mar 2014 11:44:54 +0000 Subject: Class file format specification for Java 8? In-Reply-To: References: <71108AD9-B053-44C2-9230-AEDBA187CB91@x9c.fr> Message-ID: Le 25 mars 2014 ? 10:35, Paul Sandoz a ?crit : > > > On Mar 25, 2014, at 11:10 AM, forum at x9c.fr wrote: > >> Dear List, >> >> Now that Java 8 is released, I may want to update some of my >> tools to properly support this version. To this end, I wonder if >> there is an updated version of the "class file format specification", >> or any kind of delta against the version for 1.7. >> >> Thanks in advance for any pointer. >> > > From here: > > http://docs.oracle.com/javase/specs/ > > you can get a diff of the changes to JVM spec from 7 to 8: > > http://docs.oracle.com/javase/specs/jvms/se8/jvms8-diffs.pdf > > Paul. Thanks a lot, I was not able to find it through Google. Kind regards, Xavier From marko.topolnik at gmail.com Tue Mar 25 20:46:31 2014 From: marko.topolnik at gmail.com (Marko Topolnik) Date: Tue, 25 Mar 2014 21:46:31 +0100 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: References: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> Message-ID: >> once the processing of a batch has >> commenced, after say 10-20 ms a sort of self-inspection could be >> performed by considering the count of elements processed so far. If >> there is still a lot more to do, further tasks could be split off and >> enqueued so that other threads may steal them. The size of the splits >> could be informed by the number of elements that were processed. > > We don't need this kind of thing for an ArrayList, and we might not need > it when the size or size estimate of the sequential source is known, if N > is "small" we could bias towards high-Q. We are dealing with a worst-case > scenario of a sequentially ordered source of unknown size. The trouble is, that worst case isn't at all exotic and pertains to almost any instance of the Big Data problem class, where it is not practical to load the complete input into working memory. In those cases estimating the size, even when possible, simply isn't relevant to efficient parallelization because the complete setup revolves around processing an ongoing stream of data. Compared to the Fork/Join model, which can minimize the cost of merging the results from subtasks, sequential stream processing may entail more overhead in that area, but that is an unavoidable consequence of the basic premise. > It's easy to implement a spliterator whose trySplit returns null, and an > instance of that spliterator could be wrapped in another that extracts the > parallelism. It is also easy to extend from Spliterator.AbstractSpliterator I have put in some legwork trying to follow your lead here and this is what I found out: AbstractSpliterator is indeed effective as a template to implement arbitary advancing logic, but it is ineffective as an aid to implement arbitrary spliting logic. On the other hand, I would expect a class such as BufferedReader to already provide its Spliterator which takes care of advancing, so for the work I have to do on my side, I get no help from the current framework. I have additionally realized that I can indeed get to BufferedReader's spliterator, although in a roundabout way which will do some extra work that I don't need: br.lines().spliterator() For my purpose must I ignore the stream returned from lines() and keep only its spliterator; this is perhaps something that could be improved in the API. I can then feed that spliterator into a wrapper such as the following: import static java.util.Spliterators.spliterator; import java.util.Comparator; import java.util.Spliterator; import java.util.function.Consumer; public class FixedBatchSpliteratorWrapper implements Spliterator { private final Spliterator spliterator; private final int batchSize; private final int characteristics; private long est; public FixedBatchSpliteratorWrapper(Spliterator toWrap, long est, int batchSize) { final int c = toWrap.characteristics(); this.characteristics = (c & SIZED) != 0 ? c | SUBSIZED : c; this.spliterator = toWrap; this.batchSize = batchSize; this.est = toWrap.estimateSize(); } public FixedBatchSpliteratorWrapper(Spliterator toWrap, int batchSize) { this(toWrap, toWrap.estimateSize(), batchSize); } @Override public Spliterator trySplit() { final HoldingConsumer holder = new HoldingConsumer<>(); if (!spliterator.tryAdvance(holder)) return null; final Object[] a = new Object[batchSize]; int j = 0; do a[j] = holder.value; while (++j < batchSize && tryAdvance(holder)); if (est != Long.MAX_VALUE) est -= j; return spliterator(a, 0, j, characteristics()); } @Override public boolean tryAdvance(Consumer action) { return spliterator.tryAdvance(action); } @Override public void forEachRemaining(Consumer action) { spliterator.forEachRemaining(action); } @Override public Comparator getComparator() { if (hasCharacteristics(SORTED)) return null; throw new IllegalStateException(); } @Override public long estimateSize() { return est; } @Override public int characteristics() { return characteristics; } static final class HoldingConsumer implements Consumer { Object value; @Override public void accept(T value) { this.value = value; } } } I think something of this kind, but more factored, could be a good basis for an addition to the API. I would factor this such that the client may only plug in his batch size-determining logic, without having to implement the, it must be admitted, quite hairy part where the data is spilt into the array bucket and handed over to the ArraySpliterator. -Marko From paul.sandoz at oracle.com Wed Mar 26 09:52:15 2014 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 26 Mar 2014 10:52:15 +0100 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: References: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> Message-ID: <2FAF050D-4EE1-4EBC-B6F3-C99F77C0A2DD@oracle.com> On Mar 25, 2014, at 9:46 PM, Marko Topolnik wrote: >>> once the processing of a batch has >>> commenced, after say 10-20 ms a sort of self-inspection could be >>> performed by considering the count of elements processed so far. If >>> there is still a lot more to do, further tasks could be split off and >>> enqueued so that other threads may steal them. The size of the splits >>> could be informed by the number of elements that were processed. >> >> We don't need this kind of thing for an ArrayList, and we might not need >> it when the size or size estimate of the sequential source is known, if N >> is "small" we could bias towards high-Q. We are dealing with a worst-case >> scenario of a sequentially ordered source of unknown size. > > The trouble is, that worst case isn't at all exotic Agreed, i was implying that any work done to improve this case should not affect other cases. There are many cases within j.u.c classes where we use similar techniques, but for these cases we do have a good estimate for the root size so we could potentially tweak the arithmetic progression properties. > and pertains to almost > any instance of the Big Data problem class, where it is not practical to load > the complete input into working memory. In those cases estimating the size, > even when possible, simply isn't relevant to efficient parallelization > because the complete setup revolves around processing an ongoing stream of > data. Compared to the Fork/Join model, which can minimize the cost of > merging the results from subtasks, sequential stream processing may entail > more overhead in that area, but that is an unavoidable consequence of the > basic premise. > >> It's easy to implement a spliterator whose trySplit returns null, and an >> instance of that spliterator could be wrapped in another that extracts the >> parallelism. It is also easy to extend from Spliterator.AbstractSpliterator > > I have put in some legwork trying to follow your lead here and this is what I > found out: AbstractSpliterator is indeed effective as a template to implement > arbitary advancing logic, but it is ineffective as an aid to implement > arbitrary spliting logic. Yes, you have to override the trySplit or wrap. > On the other hand, I would expect a class such as > BufferedReader to already provide its Spliterator which takes care of > advancing, so for the work I have to do on my side, I get no help from the > current framework. > We did not want to expose out spliterator directly in such a manner everywhere we expose out stream views, but i see below you have found the "escape hatch"! > I have additionally realized that I can indeed get to BufferedReader's > spliterator, although in a roundabout way which will do some extra work that I don't need: > > br.lines().spliterator() > > For my purpose must I ignore the stream returned from lines() and keep only its > spliterator; this is perhaps something that could be improved in the API. I can > then feed that spliterator into a wrapper such as the following: > > import static java.util.Spliterators.spliterator; > > import java.util.Comparator; > import java.util.Spliterator; > import java.util.function.Consumer; > > public class FixedBatchSpliteratorWrapper implements Spliterator { > private final Spliterator spliterator; > private final int batchSize; > private final int characteristics; > private long est; > > public FixedBatchSpliteratorWrapper(Spliterator toWrap, long est, int batchSize) { > final int c = toWrap.characteristics(); > this.characteristics = (c & SIZED) != 0 ? c | SUBSIZED : c; > this.spliterator = toWrap; > this.batchSize = batchSize; > this.est = toWrap.estimateSize(); > } > public FixedBatchSpliteratorWrapper(Spliterator toWrap, int batchSize) { > this(toWrap, toWrap.estimateSize(), batchSize); > } > @Override public Spliterator trySplit() { > final HoldingConsumer holder = new HoldingConsumer<>(); > if (!spliterator.tryAdvance(holder)) return null; > final Object[] a = new Object[batchSize]; > int j = 0; > do a[j] = holder.value; while (++j < batchSize && tryAdvance(holder)); > if (est != Long.MAX_VALUE) est -= j; > return spliterator(a, 0, j, characteristics()); > } > @Override public boolean tryAdvance(Consumer action) { > return spliterator.tryAdvance(action); > } > @Override public void forEachRemaining(Consumer action) { > spliterator.forEachRemaining(action); > } > @Override public Comparator getComparator() { > if (hasCharacteristics(SORTED)) return null; > throw new IllegalStateException(); > } > @Override public long estimateSize() { return est; } > @Override public int characteristics() { return characteristics; } > > static final class HoldingConsumer implements Consumer { > Object value; > @Override public void accept(T value) { this.value = value; } > } > } > > > I think something of this kind, but more factored, could be a good > basis for an addition to the API. I would factor this such that the client may > only plug in his batch size-determining logic, without having to implement the, > it must be admitted, quite hairy part where the data is spilt into the array > bucket and handed over to the ArraySpliterator. > Definitely for advanced developers. That impl could be a useful addition to spliterator utilities. It would be nice to define an AbstractFixedBatchSpliterator extending from Spliterators.AbstractSpliterator so one can choose to wrap or extend for the iteration logic, but to be useful it requires we change certain private fields to protected on Spliterators.AbstractSpliterator. Or simply expose out new constructors to AbstractSpliterator to configure the progression. I do worry that most developers will not know what the batch size should be so having something like: Files.lines(Path p, int initialBatchSize, int incrementBatchSize) could be confusing. In some cases it might be better to reformulate as a size estimate, but even so it still feels unsatisfying at this level of abstraction. Paul. From marko.topolnik at gmail.com Wed Mar 26 10:16:46 2014 From: marko.topolnik at gmail.com (Marko Topolnik) Date: Wed, 26 Mar 2014 11:16:46 +0100 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: <2FAF050D-4EE1-4EBC-B6F3-C99F77C0A2DD@oracle.com> References: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> <2FAF050D-4EE1-4EBC-B6F3-C99F77C0A2DD@oracle.com> Message-ID: <0818A698-D6A3-4CAA-84A9-10745FC83D29@gmail.com> On 26. o?u. 2014., at 10:52, Paul Sandoz wrote: > I do worry that most developers will not know what the batch size should be so having something like: > > Files.lines(Path p, int initialBatchSize, int incrementBatchSize) > > could be confusing. In some cases it might be better to reformulate as a size estimate, but even so it still feels unsatisfying at this level of abstraction. I admit to being already biased by experience, but I shall say this anyway: it feels quite easy to me to guess a good batch size. We usually know enough about our solution to be able to determine per-element processing cost at least to within two orders of magnitude, and usually even that level of imprecision is good enough. For example, if my cost is 1 microsecond, anything above 100 as a batch size will tend to work quite well. The upper bound is mostly constrained by my typical input size, but probably anything up to 10,000 cannot hurt whatever the input size. I can't help feeling that general advice to aim for 1-10 ms processing time per batch should work for most people, even if the actual cost deviates by an order of magnitude up or down. On the other hand, I agree that involving the low-level mechanics of batch size at such a prominent place in the API as an argument to Files.lines() might be dangerous, especially since there may after all be a good way to employ automatic heuristics instead of specifying from the outside. I suppose a balance has to be struck between the "happy day" and the "special needs" API complexity. Currently the imbalance is on the side of the happy day. -Marko From victor.antunes.ignacio at gmail.com Wed Mar 26 15:03:19 2014 From: victor.antunes.ignacio at gmail.com (Victor Antunes) Date: Wed, 26 Mar 2014 12:03:19 -0300 Subject: Lambda behaving differently than anonymous inner class Message-ID: Hello all, This e-mail is a follow-up to a question I've posted on StackOverflow: http://stackoverflow.com/questions/22648079/lambda-behaving-differently-than-anonymous-inner-class I'm relatively new to Java, and decided to pick up on lambda since the past few days. So I wrote a very simple anonymous inner class and wrote an equivalent lambda. However, the lambda output was different, and it very strongly appears to be a bug. Given: interface Supplier { T get(T t);} Supplier s1 = new Supplier() { @Override public Integer get(Integer t) { return t++; }};Supplier s2 = t -> t++;System.out.println(s1.get(2));System.out.println(s2.get(2)); The output is 2 and 3, NOT 2 and 2, as one would expect. More info, including discussion about bytecode is available at the SO link above. I'm also new to this list, so apologies if I've broken any mailing list etiquette. -- Kind regards, Victor Antunes From spullara at gmail.com Wed Mar 26 15:14:57 2014 From: spullara at gmail.com (Sam Pullara) Date: Wed, 26 Mar 2014 08:14:57 -0700 Subject: Lambda behaving differently than anonymous inner class In-Reply-To: References: Message-ID: <67507E82-CA68-4C2C-86D9-76F9EBDD3E8D@gmail.com> I'm not sure there is a valid use case but it looks like a bug to me. Does it reproduce if you define the lambda as { return t++; } ? Sam ---Sent from Boxer | http://getboxer.com Hello all, This e-mail is a follow-up to a question I've posted on StackOverflow: http://stackoverflow.com/questions/22648079/lambda-behaving-differently-than-anonymous-inner-class I'm relatively new to Java, and decided to pick up on lambda since the past few days. So I wrote a very simple anonymous inner class and wrote an equivalent lambda. However, the lambda output was different, and it very strongly appears to be a bug. Given: interface Supplier { T get(T t);} Supplier s1 = new Supplier() { @Override public Integer get(Integer t) { return t++; }};Supplier s2 = t -> t++;System.out.println(s1.get(2));System.out.println(s2.get(2)); The output is 2 and 3, NOT 2 and 2, as one would expect. More info, including discussion about bytecode is available at the SO link above. I'm also new to this list, so apologies if I've broken any mailing list etiquette. -- Kind regards, Victor Antunes From victor.antunes.ignacio at gmail.com Wed Mar 26 15:21:11 2014 From: victor.antunes.ignacio at gmail.com (Victor Antunes) Date: Wed, 26 Mar 2014 12:21:11 -0300 Subject: Lambda behaving differently than anonymous inner class In-Reply-To: <67507E82-CA68-4C2C-86D9-76F9EBDD3E8D@gmail.com> References: <67507E82-CA68-4C2C-86D9-76F9EBDD3E8D@gmail.com> Message-ID: Yes, using { return t++ } also reproduces the output 3. Forgot to mention that, sorry. Also, I've tested using both build build 1.8.0-b81 and build 1.8.0-b132 under Ubuntu 12.04, if that's of any interest. 2014-03-26 12:14 GMT-03:00 Sam Pullara : > I'm not sure there is a valid use case but it looks like a bug to me. Does > it reproduce if you define the lambda as { return t++; } ? > > Sam > > --- > Sent from Boxer | http://getboxer.com > On March 26, 2014 at 8:03:19 AM PDT, Victor Antunes < > victor.antunes.ignacio at gmail.com> wrote: > > Hello all, > > This e-mail is a follow-up to a question I've posted on StackOverflow: > > http://stackoverflow.com/questions/22648079/lambda-behaving-differently-than-anonymous-inner-class > > > I'm relatively new to Java, and decided to pick up on lambda since the past > few days. So I wrote a very simple anonymous inner class and wrote an > equivalent lambda. > > However, the lambda output was different, and it very strongly appears to > be a bug. > > Given: > > interface Supplier { > > T get(T t);} > > Supplier s1 = new Supplier() { > > @Override > public Integer get(Integer t) { > return t++; > }};Supplier s2 = t -> > > t++;System.out.println(s1.get(2));System.out.println(s2.get(2)); > > The output is 2 and 3, NOT 2 and 2, as one would expect. > > More info, including discussion about bytecode is available at the SO link > above. > > I'm also new to this list, so apologies if I've broken any mailing list > etiquette. > > -- > Kind regards, > > Victor Antunes > > -- att. Victor Antunes From sergey.kuksenko at oracle.com Wed Mar 26 15:25:37 2014 From: sergey.kuksenko at oracle.com (Sergey Kuksenko) Date: Wed, 26 Mar 2014 19:25:37 +0400 Subject: Lambda behaving differently than anonymous inner class In-Reply-To: <67507E82-CA68-4C2C-86D9-76F9EBDD3E8D@gmail.com> References: <67507E82-CA68-4C2C-86D9-76F9EBDD3E8D@gmail.com> Message-ID: <5332F171.3020701@oracle.com> it's javac bug. Here is bytecode of desugared lambda method: private static java.lang.Integer lambda$main$0(java.lang.Integer); Code: 0: aload_0 1: invokevirtual #9 // Method java/lang/Integer.intValue:()I 4: iconst_1 5: iadd 6: invokestatic #6 // Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer; 9: dup 10: astore_0 11: astore_1 12: aload_0 13: areturn The same bytecode if define the lambda as { return t++; } On 03/26/2014 07:14 PM, Sam Pullara wrote: > I'm not sure there is a valid use case but it looks like a bug to me. Does it reproduce if you define the lambda as { return t++; } ? Sam > > ---Sent from Boxer | http://getboxer.com > > Hello all, > > > > This e-mail is a follow-up to a question I've posted on StackOverflow: > > http://stackoverflow.com/questions/22648079/lambda-behaving-differently-than-anonymous-inner-class > > > > > > I'm relatively new to Java, and decided to pick up on lambda since the past > > few days. So I wrote a very simple anonymous inner class and wrote an > > equivalent lambda. > > > > However, the lambda output was different, and it very strongly appears to > > be a bug. > > > > Given: > > > > interface Supplier { > > > > T get(T t);} > > > > Supplier s1 = new Supplier() { > > @Override > > public Integer get(Integer t) { > > return t++; > > }};Supplier s2 = t -> > > t++;System.out.println(s1.get(2));System.out.println(s2.get(2)); > > > > The output is 2 and 3, NOT 2 and 2, as one would expect. > > > > More info, including discussion about bytecode is available at the SO link > > above. > > > > I'm also new to this list, so apologies if I've broken any mailing list > > etiquette. > > > > -- > Kind regards, > > > > Victor Antunes > > > > -- Best regards, Sergey Kuksenko From daniel.smith at oracle.com Wed Mar 26 16:56:49 2014 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 26 Mar 2014 10:56:49 -0600 Subject: Lambda behaving differently than anonymous inner class In-Reply-To: References: Message-ID: Thanks for the report. The bug is filed here: https://bugs.openjdk.java.net/browse/JDK-8038420 ?Dan On Mar 26, 2014, at 9:03 AM, Victor Antunes wrote: > Hello all, > > This e-mail is a follow-up to a question I've posted on StackOverflow: > http://stackoverflow.com/questions/22648079/lambda-behaving-differently-than-anonymous-inner-class > > > I'm relatively new to Java, and decided to pick up on lambda since the past > few days. So I wrote a very simple anonymous inner class and wrote an > equivalent lambda. > > However, the lambda output was different, and it very strongly appears to > be a bug. > > Given: > > interface Supplier { > > T get(T t);} > > Supplier s1 = new Supplier() { > @Override > public Integer get(Integer t) { > return t++; > }};Supplier s2 = t -> > t++;System.out.println(s1.get(2));System.out.println(s2.get(2)); > > The output is 2 and 3, NOT 2 and 2, as one would expect. > > More info, including discussion about bytecode is available at the SO link > above. > > I'm also new to this list, so apologies if I've broken any mailing list > etiquette. > > -- > Kind regards, > > Victor Antunes > From forax at univ-mlv.fr Wed Mar 26 17:48:18 2014 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 26 Mar 2014 18:48:18 +0100 Subject: Lambda behaving differently than anonymous inner class In-Reply-To: References: Message-ID: <533312E2.5090304@univ-mlv.fr> On 03/26/2014 05:56 PM, Dan Smith wrote: > Thanks for the report. The bug is filed here: > > https://bugs.openjdk.java.net/browse/JDK-8038420 > > ?Dan Yes, definitively a bug ! Victor, modify a parameter inside a lambda is something weird|.|| Suppliers2 =t ->t++;| means | Suppliers2 =t ->{ t = t + 1; return t; }; |so this is equivalent to| Suppliers2 =t ->t+ 1;| cheers, R?mi > > On Mar 26, 2014, at 9:03 AM, Victor Antunes wrote: > >> Hello all, >> >> This e-mail is a follow-up to a question I've posted on StackOverflow: >> http://stackoverflow.com/questions/22648079/lambda-behaving-differently-than-anonymous-inner-class >> >> >> I'm relatively new to Java, and decided to pick up on lambda since the past >> few days. So I wrote a very simple anonymous inner class and wrote an >> equivalent lambda. >> >> However, the lambda output was different, and it very strongly appears to >> be a bug. >> >> Given: >> >> interface Supplier { >> >> T get(T t);} >> >> Supplier s1 = new Supplier() { >> @Override >> public Integer get(Integer t) { >> return t++; >> }};Supplier s2 = t -> >> t++;System.out.println(s1.get(2));System.out.println(s2.get(2)); >> >> The output is 2 and 3, NOT 2 and 2, as one would expect. >> >> More info, including discussion about bytecode is available at the SO link >> above. >> >> I'm also new to this list, so apologies if I've broken any mailing list >> etiquette. >> >> -- >> Kind regards, >> >> Victor Antunes >> > From spullara at gmail.com Wed Mar 26 17:50:46 2014 From: spullara at gmail.com (Sam Pullara) Date: Wed, 26 Mar 2014 10:50:46 -0700 Subject: Lambda behaving differently than anonymous inner class In-Reply-To: <533312E2.5090304@univ-mlv.fr> References: <533312E2.5090304@univ-mlv.fr> Message-ID: <4261F2B7-8BF1-4FD1-8F3C-B84B15E4BF0D@gmail.com> I think there was some discussion about whether the lambda parameters should be final. I still think they should have been.? Sam ---Sent from Boxer | http://getboxer.com On 03/26/2014 05:56 PM, Dan Smith wrote: > Thanks for the report. The bug is filed here: > > https://bugs.openjdk.java.net/browse/JDK-8038420 > > ?Dan Yes, definitively a bug ! Victor, modify a parameter inside a lambda is something weird|.|| Suppliers2 =t ->t++;| means | Suppliers2 =t ->{ t = t + 1; return t; }; |so this is equivalent to| Suppliers2 =t ->t+ 1;| cheers, R?mi > > On Mar 26, 2014, at 9:03 AM, Victor Antunes wrote: > >> Hello all, >> >> This e-mail is a follow-up to a question I've posted on StackOverflow: >> http://stackoverflow.com/questions/22648079/lambda-behaving-differently-than-anonymous-inner-class >> >> >> I'm relatively new to Java, and decided to pick up on lambda since the past >> few days. So I wrote a very simple anonymous inner class and wrote an >> equivalent lambda. >> >> However, the lambda output was different, and it very strongly appears to >> be a bug. >> >> Given: >> >> interface Supplier { >> >> T get(T t);} >> >> Supplier s1 = new Supplier() { >> @Override >> public Integer get(Integer t) { >> return t++; >> }};Supplier s2 = t -> >> t++;System.out.println(s1.get(2));System.out.println(s2.get(2)); >> >> The output is 2 and 3, NOT 2 and 2, as one would expect. >> >> More info, including discussion about bytecode is available at the SO link >> above. >> >> I'm also new to this list, so apologies if I've broken any mailing list >> etiquette. >> >> -- >> Kind regards, >> >> Victor Antunes >> > From forax at univ-mlv.fr Wed Mar 26 18:40:37 2014 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 26 Mar 2014 19:40:37 +0100 Subject: Lambda behaving differently than anonymous inner class In-Reply-To: <4261F2B7-8BF1-4FD1-8F3C-B84B15E4BF0D@gmail.com> References: <533312E2.5090304@univ-mlv.fr> <4261F2B7-8BF1-4FD1-8F3C-B84B15E4BF0D@gmail.com> Message-ID: <53331F25.1080100@univ-mlv.fr> On 03/26/2014 06:50 PM, Sam Pullara wrote: > I think there was some discussion about whether the lambda parameters > should be final. I still think they should have been. > > Sam Yes, but method parameters should have been final by default too. IDE should mark every parameters final by default. > > --- > Sent from Boxer | http://getboxer.com and stop sending email from boxer, there is still no Android version :) R?mi > On March 26, 2014 at 10:48:18 AM PDT, Remi Forax > wrote: >> On 03/26/2014 05:56 PM, Dan Smith wrote: >> > Thanks for the report. The bug is filed here: >> > >> > https://bugs.openjdk.java.net/browse/JDK-8038420 >> > >> > ?Dan >> >> Yes, definitively a bug ! >> >> Victor, modify a parameter inside a lambda is something weird|.|| >> Suppliers2 =t ->t++;| >> means >> | Suppliers2 =t ->{ >> t = t + 1; >> return t; >> }; >> |so this is equivalent to| >> Suppliers2 =t ->t+ 1;| >> >> cheers, >> R?mi >> >> > >> > On Mar 26, 2014, at 9:03 AM, Victor Antunes wrote: >> > >> >> Hello all, >> >> >> >> This e-mail is a follow-up to a question I've posted on StackOverflow: >> >> >> http://stackoverflow.com/questions/22648079/lambda-behaving-differently-than-anonymous-inner-class >> >> >> >> >> >> I'm relatively new to Java, and decided to pick up on lambda since >> the past >> >> few days. So I wrote a very simple anonymous inner class and wrote an >> >> equivalent lambda. >> >> >> >> However, the lambda output was different, and it very strongly >> appears to >> >> be a bug. >> >> >> >> Given: >> >> >> >> interface Supplier{ >> >> >> >> T get(T t);} >> >> >> >> Suppliers1 = new Supplier() { >> >> @Override >> >> public Integer get(Integer t) { >> >> return t++; >> >> }};Suppliers2 = t -> >> >> t++;System.out.println(s1.get(2));System.out.println(s2.get(2)); >> >> >> >> The output is 2 and 3, NOT 2 and 2, as one would expect. >> >> >> >> More info, including discussion about bytecode is available at the >> SO link >> >> above. >> >> >> >> I'm also new to this list, so apologies if I've broken any mailing >> list >> >> etiquette. >> >> >> >> -- >> >> Kind regards, >> >> >> >> Victor Antunes >> >> >> > >> >> From zhong.j.yu at gmail.com Wed Mar 26 19:31:35 2014 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Wed, 26 Mar 2014 14:31:35 -0500 Subject: Lambda behaving differently than anonymous inner class In-Reply-To: <4261F2B7-8BF1-4FD1-8F3C-B84B15E4BF0D@gmail.com> References: <533312E2.5090304@univ-mlv.fr> <4261F2B7-8BF1-4FD1-8F3C-B84B15E4BF0D@gmail.com> Message-ID: At least, *implicit* lambda parameters should be final. But it's too late now. On Wed, Mar 26, 2014 at 12:50 PM, Sam Pullara wrote: > I think there was some discussion about whether the lambda parameters should be final. I still think they should have been. Sam > > ---Sent from Boxer | http://getboxer.com > > On 03/26/2014 05:56 PM, Dan Smith wrote: > >> Thanks for the report. The bug is filed here: > >> > >> https://bugs.openjdk.java.net/browse/JDK-8038420 > >> > >> --Dan > > > > Yes, definitively a bug ! > > > > Victor, modify a parameter inside a lambda is something weird|.|| > > Suppliers2 =t ->t++;| > > means > > | Suppliers2 =t ->{ > > t = t + 1; > > return t; > > }; > > |so this is equivalent to| > > Suppliers2 =t ->t+ 1;| > > > > cheers, > > R?mi > > > >> > >> On Mar 26, 2014, at 9:03 AM, Victor Antunes wrote: > >> > >>> Hello all, > >>> > >>> This e-mail is a follow-up to a question I've posted on StackOverflow: > >>> http://stackoverflow.com/questions/22648079/lambda-behaving-differently-than-anonymous-inner-class > >>> > >>> > >>> I'm relatively new to Java, and decided to pick up on lambda since the past > >>> few days. So I wrote a very simple anonymous inner class and wrote an > >>> equivalent lambda. > >>> > >>> However, the lambda output was different, and it very strongly appears to > >>> be a bug. > >>> > >>> Given: > >>> > >>> interface Supplier { > >>> > >>> T get(T t);} > >>> > >>> Supplier s1 = new Supplier() { > >>> @Override > >>> public Integer get(Integer t) { > >>> return t++; > >>> }};Supplier s2 = t -> > >>> t++;System.out.println(s1.get(2));System.out.println(s2.get(2)); > >>> > >>> The output is 2 and 3, NOT 2 and 2, as one would expect. > >>> > >>> More info, including discussion about bytecode is available at the SO link > >>> above. > >>> > >>> I'm also new to this list, so apologies if I've broken any mailing list > >>> etiquette. > >>> > >>> -- >>> Kind regards, > >>> > >>> Victor Antunes > >>> > >> > > > > > > From zhong.j.yu at gmail.com Wed Mar 26 19:32:57 2014 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Wed, 26 Mar 2014 14:32:57 -0500 Subject: Lambda behaving differently than anonymous inner class In-Reply-To: References: Message-ID: Another example: Consumer c = t -> System.out.println(t++); c.accept(2); no no no no... Zhong Yu On Wed, Mar 26, 2014 at 10:03 AM, Victor Antunes wrote: > Hello all, > > This e-mail is a follow-up to a question I've posted on StackOverflow: > http://stackoverflow.com/questions/22648079/lambda-behaving-differently-than-anonymous-inner-class > > > I'm relatively new to Java, and decided to pick up on lambda since the past > few days. So I wrote a very simple anonymous inner class and wrote an > equivalent lambda. > > However, the lambda output was different, and it very strongly appears to > be a bug. > > Given: > > interface Supplier { > > T get(T t);} > > Supplier s1 = new Supplier() { > @Override > public Integer get(Integer t) { > return t++; > }};Supplier s2 = t -> > t++;System.out.println(s1.get(2));System.out.println(s2.get(2)); > > The output is 2 and 3, NOT 2 and 2, as one would expect. > > More info, including discussion about bytecode is available at the SO link > above. > > I'm also new to this list, so apologies if I've broken any mailing list > etiquette. > > -- > Kind regards, > > Victor Antunes > From babanin at gmail.com Wed Mar 26 21:22:18 2014 From: babanin at gmail.com (Ivan Babanin) Date: Thu, 27 Mar 2014 01:22:18 +0400 Subject: Lambda behaving differently than anonymous inner class Message-ID: <000001cf4939$78c147d0$6a43d770$@gmail.com> It's definitely a javac bug. Anonymous class and lambda converts to following intermediate representations: @Override() public Integer get(Integer t) { return (let /*synthetic*/ final Integer $112619572 = t in (let /*synthetic*/ final Integer $1295226194 = t = Integer.valueOf((int)(t.intValue() + 1)) in $112619572)); } /*synthetic*/ private static Integer lambda$main$0(final Integer t) { return (let /*synthetic*/ final Integer $1146147158 = t = Integer.valueOf((int)(t.intValue() + 1)) in t); } In lambda generated method parameter marked as final, because LambdaToMethod translator marks all parameters as FINAL (according source code LambdaTranslationContext.translate(.) : 1899). Then let expression builder checks variable flags and when if it's final omits temporary variable generation (according source code Lower. abstractRval(.) : 2277), because modification considered to be prohibited. Possible solutions: 1) Forbid parameter modification inside lambda or 2) Remove FINAL flag from local variable (LambdaTranslationContext.translate(.) : 1894) and parameter (LambdaTranslationContext.translate(.) : 1899) in lamda generated method: case LOCAL_VAR: ret = new VarSymbol(FINAL, name, types.erasure(sym.type), translatedSym); ... case PARAM: ret = new VarSymbol(FINAL | PARAMETER, name, types.erasure(sym.type), translatedSym); ... I removed FINAL flag and got expected results on tests from: https://bugs.openjdk.java.net/browse/JDK-8038420 sy, Ivan From wupuyuan at gmail.com Fri Mar 28 02:40:05 2014 From: wupuyuan at gmail.com (=?UTF-8?B?5ZC055Ke5riK?=) Date: Fri, 28 Mar 2014 10:40:05 +0800 Subject: How about the performance of lambda for more cores Message-ID: Dear: "State of the Lambda" introduce that the Lambda project is working for parallel APIs, now how to test? I made a comparison between anonymous inner class and lambda expression, the result is lambda is not so good as anonymous inner class in one thread. So, how about concurrent program, and if it's also depends on "invokedynamic" which is not fast enough? -- B&R From spullara at gmail.com Fri Mar 28 03:14:25 2014 From: spullara at gmail.com (Sam Pullara) Date: Thu, 27 Mar 2014 20:14:25 -0700 Subject: How about the performance of lambda for more cores In-Reply-To: References: Message-ID: It would be great if your benchmark was on github (or somewhere else public) so that we could see the particular case you are testing. Sam On Thu, Mar 27, 2014 at 7:40 PM, ??? wrote: > Dear: > "State of the Lambda" introduce that the Lambda project is working > for parallel APIs, now how to test? > I made a comparison between anonymous inner class and lambda > expression, the result is lambda is not so good as anonymous inner class in > one thread. > So, how about concurrent program, and if it's also depends on > "invokedynamic" which is not > fast enough? > -- > B&R > > From forax at univ-mlv.fr Fri Mar 28 07:53:29 2014 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 28 Mar 2014 08:53:29 +0100 Subject: How about the performance of lambda for more cores In-Reply-To: References: Message-ID: <53352A79.7050703@univ-mlv.fr> On 03/28/2014 03:40 AM, ??? wrote: > Dear: > "State of the Lambda" introduce that the Lambda project is working > for parallel APIs, now how to test? > I made a comparison between anonymous inner class and lambda > expression, the result is lambda is not so good as anonymous inner class in > one thread. > So, how about concurrent program, and if it's also depends on > "invokedynamic" which is not > fast enough? invokedynamic is used to create the lambda not to call it, so usually it's not the bottleneck. cheers, R?mi From paul.sandoz at oracle.com Fri Mar 28 08:20:57 2014 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 28 Mar 2014 09:20:57 +0100 Subject: How about the performance of lambda for more cores In-Reply-To: References: Message-ID: <28D4E688-CBBD-4C6C-95A7-A84604312F44@oracle.com> On Mar 28, 2014, at 4:14 AM, Sam Pullara wrote: > It would be great if your benchmark was on github (or somewhere else > public) so that we could see the particular case you are testing. > +1 Micro-benchmarking is tricky to get right. The results of our measurements were that lambda's are generally as good as or faster than inner classes, depending on the context. Of relevance is Sergey's Lambda Performance talk @ JVMLS 2013: http://www.oracle.com/technetwork/java/javase/community/jvmls2013-2013900.html http://www.oracle.com/technetwork/java/jvmls2013kuksen-2014088.pdf http://medianetwork.oracle.com/video/player/2623576348001 IIRC based on those results improvements were made to the lambda meta-fractory code. Paul. From sergey.kuksenko at oracle.com Fri Mar 28 12:42:08 2014 From: sergey.kuksenko at oracle.com (Sergey Kuksenko) Date: Fri, 28 Mar 2014 16:42:08 +0400 Subject: How about the performance of lambda for more cores In-Reply-To: References: Message-ID: <53356E20.60007@oracle.com> On 03/28/2014 06:40 AM, ??? wrote: > Dear: > "State of the Lambda" introduce that the Lambda project is working > for parallel APIs, now how to test? It would be better know what exactly do want to test? "Testing parallel API" is too general, let's start from small pieces. > I made a comparison between anonymous inner class and lambda > expression, the result is lambda is not so good as anonymous inner class in > one thread. I'd like to see your results, how did you do measurement and etc. Your conclusion "not so good" don't bring us any information. Which parts of lambda life cycle did you measure? > So, how about concurrent program, and if it's also depends on > "invokedynamic" which is not > fast enough? What does it mean "fast enough" if we are talking about invokedynamic? Again, which parts of lambda life cycle do you want evaluate in case of concurrent program? Don't forget, invokedynamic is used only for lambda linkage, indy isn't used neither by lambda capturing nor by lambda invocation. -- Best regards, Sergey Kuksenko From Donald.Raab at gs.com Fri Mar 28 20:57:21 2014 From: Donald.Raab at gs.com (Raab, Donald) Date: Fri, 28 Mar 2014 16:57:21 -0400 Subject: GS Collections 5.0 released and lambda ready Message-ID: <6712820CB52CFB4D842561213A77C054053E72E92A@GSCMAMP09EX.firmwide.corp.gs.com> Hi All, We released GS Collections 5.0 this week on GitHub and Maven Central. In addition to a host of new apis and features, we have also converted our test case modules to leverage Java 8. We have converted nearly all of our anonymous inner classes in our tests to lambdas and method references. In total, we run around 118,000 tests as part of our unit test suite note. We saw a total reduction of around 8% of the lines of code in our test module so far. We will be continuing to introduce more lambda and method reference usages in our upcoming releases. To browse our tests and lambda usages in GitHub, you can use this link: https://github.com/goldmansachs/gs-collections/tree/master/unit-tests/src/test/java/com/gs/collections/impl The GSC framework itself is still compatible with Java 5 - 8, but we wanted to give Java 8 lambdas and method references a large scale test in a code base that can benefit heavily from lambdas. We also updated our reference guide and kata training materials to include examples and training using Java 8 lambda expressions and method references. Our code, kata, release notes, documentation and couple of articles on GSC are available at the links below, and the binaries are available in Maven Central: https://github.com/goldmansachs/gs-collections https://github.com/goldmansachs/gs-collections-kata https://github.com/goldmansachs/gs-collections/releases https://github.com/goldmansachs/gs-collections/wiki Thanks, Don The Goldman Sachs Group, Inc. All rights reserved. See http://www.gs.com/disclaimer/global_email for important risk disclosures, conflicts of interest and other terms and conditions relating to this e-mail and your reliance on information contained in it. This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.gs.com/disclaimer/email for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From radhakrishnan.mohan at gmail.com Sat Mar 29 02:44:45 2014 From: radhakrishnan.mohan at gmail.com (Mohan Radhakrishnan) Date: Sat, 29 Mar 2014 08:14:45 +0530 Subject: How about the performance of lambda for more cores In-Reply-To: <53356E20.60007@oracle.com> References: <53356E20.60007@oracle.com> Message-ID: Hi, Can JMH help here ? One more benchmarking result I found. http://www.optaplanner.org/blog/2014/03/20/HowMuchFasterIsJava8.html Thanks, Mohan On Fri, Mar 28, 2014 at 6:12 PM, Sergey Kuksenko wrote: > On 03/28/2014 06:40 AM, ??? wrote: > >> Dear: >> "State of the Lambda" introduce that the Lambda project is working >> for parallel APIs, now how to test? >> > It would be better know what exactly do want to test? "Testing parallel > API" is too general, let's start from small pieces. > > > I made a comparison between anonymous inner class and lambda >> expression, the result is lambda is not so good as anonymous inner class >> in >> one thread. >> > > I'd like to see your results, how did you do measurement and etc. Your > conclusion "not so good" don't bring us any information. > Which parts of lambda life cycle did you measure? > > > So, how about concurrent program, and if it's also depends on >> "invokedynamic" which is not >> fast enough? >> > > What does it mean "fast enough" if we are talking about invokedynamic? > Again, which parts of lambda life cycle do you want evaluate in case of > concurrent program? Don't forget, invokedynamic is used only for lambda > linkage, indy isn't used neither by lambda capturing nor by lambda > invocation. > > > -- > Best regards, > Sergey Kuksenko > > From glen at organicdesign.org Tue Mar 25 12:27:19 2014 From: glen at organicdesign.org (Glen Peterson) Date: Tue, 25 Mar 2014 12:27:19 -0000 Subject: Exploiting concurrency with IteratorSpliterator of unknown size In-Reply-To: References: <9DDF65B9-1A87-4C29-B4F3-BE6B67E77A51@oracle.com> Message-ID: "There is also a tricker more general problem: that of automatically working out when to go parallel or sequential." I've been puzzling over a similar question: Who has the necessary information to decide the proper number of threads for a task? The proper number of threads might be based on: 1. The priority of the job 2. The number of physical processors 3. The expected workload of those processors while the job will be running (what other jobs are likely to run simultaneously and their priorities) 4. The processing characteristics of the job itself It seems to me that a collection or transformation only has partial information about #4 until it has run for a while and can measure itself (which is an interesting idea). Really, the operating system, or the calling code is in a much better position to make decisions about the proper number of threads to use. The simplest solution I could see would be to let the calling code pass a suggested maximum number of threads to use. The collection/transform might use less than that (or perform processing in the main thread if there is only a single item to transform). The start and measure approach to this problem might provide hints about #2 and #4 plus maybe some of #3, but completely leaves out #1. Only letting the client pick the number of threads can reasonably cover all 4 because #3 has to do with expectations, and the designer of the program knows best what's likely to happen next. I'm not sure if the JVM provides information about the number of processors or the current load, but that information could be provided manually. If you use a virtual machine or otherwise share hardware, only the operating system may be able to guess at #3, or maybe no-one has complete information. I work a lot with web apps, and if a job is big enough to benefit from running on multiple processors, I don't want it to hog the system to the point that the next user who clicks on the app has to wait for the previous user's job to finish. I recently coded up some simple tests on a 2 processor 64-bit Ubuntu system (not using Java collections) and both processors were over 95% busy even when I didn't start a second thread. I don't know yet if that's the garbage collector, or some other part of the JVM or some other cleverness happening on the other processor, but my timings with CPU/Memory-fetch-bound concurrency on my machine showed the single-threaded solution to be equal or 20% faster than a two-thread version. If that's typical, another question would be, "under what situations does using more threads speed processing?" I think that question stands even if my example does not. Maybe that makes #5. On Tue, Mar 25, 2014 at 6:06 AM, Paul Sandoz wrote: > > On Mar 24, 2014, at 9:58 PM, Marko Topolnik wrote: > >>> Yes, that is one reason why we did not base Spliterator on Iterator. We did >>> not have a compelling use-case to separate out the iteration from splitting. >>> ... >>> I get what you are saying but that is just a Spliterator that does not split :-) >> >> Seems that here we've talked past each other... my message was that, >> for the same reason that the IteratorSpliterator is useful now, a >> partial Spliterator interface (itself _not_ a Spliterator) which >> handles just the iteration aspect would be useful as a building block >> for the full Spliterator, and better than an Iterator due to the saner >> API (hasNext+next in one method, efficient iteration over all >> elements). So if all the relevant objects, such as BufferedReader in >> our case, provided that partial implementation in addition to the >> finished, defaults-based Stream, it would be a good basis for a custom >> Spliterator. >> > > I am saying we have all the building blocks to do that today, but perhaps not quite as clean as you indicate. > > It's easy to implement a spliterator whose trySplit returns null, and an instance of that spliterator could be wrapped in another that extracts the parallelism. It is also easy to extend from Spliterator.AbstractSpliterator: > > public Stream lines() { > Spliterator sp = new Spliterators.AbstractSpliterator( > Long.MAX_VALUE, Spliterator.ORDERED | Spliterator.NONNULL) { > @Override > public boolean tryAdvance(Consumer action) { > try { > String line = readLine(); > if (line != null) { > action.accept(line); > return true; > } > else { > return false; > } > } catch (IOException e) { > throw new UncheckedIOException(e); > } > } > > @Override > public void forEachRemaining(Consumer action) { > try { > String line; > while ((line = readLine()) != null) { > action.accept(line); > } > } catch (IOException e) { > throw new UncheckedIOException(e); > } > } > }; > return StreamSupport.stream(sp, false); > } > > I suspect we did not do that at the time because Spliterators.AbstractSpliterator may not have been in the API when BufferedReader.lines() was written. > > >>>> Generally, how do the "recursive decomposition" mechanics >>>> play >>>> out in our scenario here? >>>> >>> >>> We are getting into gritty details of the implementation... >>> >>> If a size estimate of the root spliterator is known then a threshold is >>> calculated so that a spliterator is no longer split if it's size estimate is below >>> that threshold. For something like an ArrayList this will create a balanced >>> tree where the leaf nodes are in proportion >>> to the targeted parallelism level of the fork/join common pool. >>> >>> In your situation it is a worst case scenario :-) a sequential source of >>> unknown size, This will create a right balanced tree, each left leaf node is a >>> prefix of elements copied into an array. Because the size is unknown the >>> Spliterator wrapping that array will not be split. Splitting of the right tree >>> nodes will occur until a null is returned (i.e. when there are no more >>> elements). Reduction operations can perform poorly on such trees due to >>> the imbalance. >>> >>> Generally the technique of splitting from a sequential source is to try and >>> ramp up quickly extracting out some work to warm things up while in parallel >>> further splitting (safely) from the source. As you have found out YMMV. >> >> I see; the premise behind this is that extracting an item from the >> sequential source has the same order of cost as its processing. As >> soon as processing cost ovewhelms reading cost, ramp-up loses >> significance because the threadpool's queue can be quickly built up by >> reading into ArraySpliterators. Therefore the story applies only to >> the contact area between two ranges of problems: those with I/O as the >> bottleneck and those with CPU as the bottleneck. The former is clearly >> not of interest here, but in the latter I see a different story >> explaining the effectiveness of the ramp-up strategy: it strikes a >> balance between the needs of high-Q and low-Q problems. For both >> classes there is a lower bound for N at which this strategy shows >> satisfactory performance, that N being a function of the batch size >> increment. With my problem, I unfortunately find myself below that >> threshold. > > Yes, unfortunately so. > > Originally we used an arithmetic progression with a difference of 1, but this could create unduly large trees resulting in performance issues when reduced/collected. > > The original stream F/J code (see AbstractTask) was too aggressive splitting for such cases and Doug solved this by a rather clever but simple solution of alternating forking and computation of the left and right tasks, which meant the initiating invocation thread was no longer aggressively spawning left nodes as fast as it could (and potentially using up lots of memory). > > There is definitely room for improvement here. > > >> >> I still seem to be missing an important piece of the puzzle in my >> understanding of F/J, so perhaps you can clear this up for me: it >> currently seems to me that, once the processing of a batch has >> commenced, after say 10-20 ms a sort of self-inspection could be >> performed by considering the count of elements processed so far. If >> there is still a lot more to do, further tasks could be split off and >> enqueued so that other threads may steal them. The size of the splits >> could be informed by the number of elements that were processed. I >> would like to know whether this is a) impossible, b) possible but >> would be ineffective, c) possible and potentially effective, but not >> yet considered for implementation, d) something else entirely? > > I doubt it is impossible, but I don't know how effective it might be until one gets the results of such an experiment. > > We don't need this kind of thing for an ArrayList, and we might not need it when the size or size estimate of the sequential source is known, if N is "small" we could bias towards high-Q. We are dealing with a worst-case scenario of a sequentially ordered source of unknown size. > > Some simple holder of a count e.g. LongAdder or AtomicLong could be updated when each leaf node completes, plus there might be stats from the F/J pool that could be used. We don't even need to have an accurate count, perhaps an estimate (size estimate from the spliterator) is sufficient. An estimate is cheaper since we can more easily leverage forEachRemaining without wrapping with a local counting consumer. > > A tricky aspect would be ensuring this statistical approach is bound with the particular style of source as we don't want to hard code this in. > > There is also a tricker more general problem: that of automatically working out when to go parallel or sequential. I think that requires gathering statistics over multiple executions of the pipeline from which it is decided to switch modes on subsequent executions (and more generally this could also be applied to optimizing the pipeline, compiling to byte code and/or executing on say a GPU). > > >> I >> remember reading a similar idea in this or the EG mailing list, but it >> was left sort of hanging in the air. >> > > Quite possibly, i don't recall such a discussion :-) > > Paul. > -- Glen K. Peterson (828) 393-0081