From Vladimir.Zakharov at gs.com Thu Aug 1 11:19:33 2013 From: Vladimir.Zakharov at gs.com (Zakharov, Vladimir) Date: Thu, 1 Aug 2013 14:19:33 -0400 Subject: GS Collections Memory Usage Comparisons Message-ID: Hello, We thought this may be something of interest to this group: http://www.goldmansachs.com/gs-collections/presentations/GSC_Memory_Tests.pdf The charts show memory metrics for different collection frameworks using simple benchmarks. All the relevant sources are available on GitHub, and the numbers should be easily reproducible. We compared JDK 7, Google Guava (currently using 14.0.1), Trove (currently using 3.0.3) and GS Collections (currently 3.2). Thank you, Don, Vlad The Goldman Sachs Group, Inc. All rights reserved. See http://www.gs.com/disclaimer/global_email for important risk disclosures, conflicts of interest and other terms and conditions relating to this e-mail and your reliance on information contained in it. This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.gs.com/disclaimer/email for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130801/a0a88e10/attachment.html From brian.goetz at oracle.com Fri Aug 2 16:39:31 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 2 Aug 2013 16:39:31 -0700 Subject: Revisiting MayHoldCloseableResource Message-ID: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> There seems to be consensus that the current strawman for MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. Flaw #1 relates to @HoldsResource. While there seems to be consensus that annotations are reasonable for imparting information like this which is suited to static analysis tools, the current proposal (a) confusingly leaks through into subclass Javadoc and (b) seems too weak. Conclusion: just dropping @HR from the current scheme (leaving only MHCR <: AC) is an improvement; we can revisit what the correct annotations for hinting at resourceful-ness later. Flaw #2 relates to MHCR. The real problem is that AutoCloseable is overspecified; it says "a resource that must be closed." This is an overreach; what j.l.AC should say is "has a close method which is called automatically on exit from a TWR block." Because of this overspecify, subclassing doesn't help, because if an AC must be closed, so must a MHCR. Conclusion: dropping MHCR and just having BaseStream <: AC is an improvement. Flaw #3 relates to the spec of AC. The conclusion of the core library maintainers is that it is acceptable to correct the spec, and replace the "must" with "may". This leaves us with a fixed AC, and BaseStream <: AC. This seems strictly better than where we are now. Let's call this "Option 1." But, it is possible we might still choose to punt further for now. Observation: the need for Stream to have a close() method comes about because of the desire to have static factory methods that will create streams that encapsulate a resource that requires closing, when there is no way to get access to that underlying resource. Such as Files.walk(dir) or Files.lines(path). For other cases, like BufferedReader.lines(), the BR itself is AC, so users can handle resource management like: try (BufferedReader b = ...) { b.lines()... } and there is no need to close the *stream* at all, since the user has access to the actual closeable resource. It is only the static methods such as Files.list(dir), which don't provide the user access in this way. The methods we provide are: Files.list(Path) Files.walk(Path, maxDepth, options) Files.walk(Path, options) Files.find(Path, maxDepth, BiPredicate, options) Files.lines(Path) Colin @ Google pointed out that if the spliterators for these streams were lazy (didn't open the resource until the terminal began), and all terminal ops called the close hook in a finally block after processing, then we wouldn't need to be AC, since there would be no way to leak a closeable resource. (This would move resource-not-found exceptions from the factory method to the terminal op.) Still, having a close() method and *not* being AC may look weird to the users. Let's call this "Option 2": have a close() method, don't extend AC, make sure that all terminals call close, make sure that all static resource-holding stream factories produce late-binding spliterators. Option 3 would be to simply get rid of these methods (note that Files.list and Files.lines are pretty simple wrappers around existing AC abstractions, DirectoryStream and BufferedReader.) Orthogonal to Options 2 and 3 would be fixing the spec of AC. From andrey.breslav at jetbrains.com Sat Aug 3 03:10:38 2013 From: andrey.breslav at jetbrains.com (Andrey Breslav) Date: Sat, 3 Aug 2013 12:10:38 +0200 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> Message-ID: <253D6820-DDDA-446B-9583-798478F0FD6E@jetbrains.com> In case you are asking for opinions, I think * we'd better fix the AC spec early than late; * have streams implement (fixed) AC (Option 1) looks better to me than just have close() (Option 2) or not having the Files.* methods (Option 3). -- Andrey Breslav http://jetbrains.com Develop with pleasure! On Aug 3, 2013, at 01:39 , Brian Goetz wrote: > There seems to be consensus that the current strawman for MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. > > Flaw #1 relates to @HoldsResource. While there seems to be consensus that annotations are reasonable for imparting information like this which is suited to static analysis tools, the current proposal (a) confusingly leaks through into subclass Javadoc and (b) seems too weak. Conclusion: just dropping @HR from the current scheme (leaving only MHCR <: AC) is an improvement; we can revisit what the correct annotations for hinting at resourceful-ness later. > > Flaw #2 relates to MHCR. The real problem is that AutoCloseable is overspecified; it says "a resource that must be closed." This is an overreach; what j.l.AC should say is "has a close method which is called automatically on exit from a TWR block." Because of this overspecify, subclassing doesn't help, because if an AC must be closed, so must a MHCR. Conclusion: dropping MHCR and just having BaseStream <: AC is an improvement. > > Flaw #3 relates to the spec of AC. The conclusion of the core library maintainers is that it is acceptable to correct the spec, and replace the "must" with "may". This leaves us with a fixed AC, and BaseStream <: AC. This seems strictly better than where we are now. Let's call this "Option 1." > > But, it is possible we might still choose to punt further for now. > > Observation: the need for Stream to have a close() method comes about because of the desire to have static factory methods that will create streams that encapsulate a resource that requires closing, when there is no way to get access to that underlying resource. Such as Files.walk(dir) or Files.lines(path). > > For other cases, like BufferedReader.lines(), the BR itself is AC, so users can handle resource management like: > > try (BufferedReader b = ...) { > b.lines()... > } > > and there is no need to close the *stream* at all, since the user has access to the actual closeable resource. It is only the static methods such as Files.list(dir), which don't provide the user access in this way. The methods we provide are: > > Files.list(Path) > Files.walk(Path, maxDepth, options) > Files.walk(Path, options) > Files.find(Path, maxDepth, BiPredicate, options) > Files.lines(Path) > > Colin @ Google pointed out that if the spliterators for these streams were lazy (didn't open the resource until the terminal began), and all terminal ops called the close hook in a finally block after processing, then we wouldn't need to be AC, since there would be no way to leak a closeable resource. (This would move resource-not-found exceptions from the factory method to the terminal op.) > > Still, having a close() method and *not* being AC may look weird to the users. > > Let's call this "Option 2": have a close() method, don't extend AC, make sure that all terminals call close, make sure that all static resource-holding stream factories produce late-binding spliterators. > > Option 3 would be to simply get rid of these methods (note that Files.list and Files.lines are pretty simple wrappers around existing AC abstractions, DirectoryStream and BufferedReader.) > > Orthogonal to Options 2 and 3 would be fixing the spec of AC. > > From joe.bowbeer at gmail.com Sat Aug 3 07:54:33 2013 From: joe.bowbeer at gmail.com (Joe Bowbeer) Date: Sat, 3 Aug 2013 07:54:33 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <253D6820-DDDA-446B-9583-798478F0FD6E@jetbrains.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <253D6820-DDDA-446B-9583-798478F0FD6E@jetbrains.com> Message-ID: Brian, As you wrote, BufferedReader.lines() gets this right and I'm thankful enough for that: try (BufferedReader r = Files.newBufferedReader(path, Charset.defaultCharset())) { anagrams(r.lines()).forEach(System.out::println); } There's a similar story with DirectoryStream versus Files.list(), I assume. Why aren't there similar formulations for Files.walk and Files.find? I'm not convinced that Stream itself should have a close() method; and if that is troubling to Files.walk, then it can take a hike. Fixing the AC spec seems fine, but not relevant to Stream. --Joe On Sat, Aug 3, 2013 at 3:10 AM, Andrey Breslav wrote: > In case you are asking for opinions, I think > * we'd better fix the AC spec early than late; > * have streams implement (fixed) AC (Option 1) looks better to me than > just have close() (Option 2) or not having the Files.* methods (Option 3). > > -- > Andrey Breslav > http://jetbrains.com > Develop with pleasure! > > > On Aug 3, 2013, at 01:39 , Brian Goetz wrote: > > > There seems to be consensus that the current strawman for > MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. > > > > Flaw #1 relates to @HoldsResource. While there seems to be consensus > that annotations are reasonable for imparting information like this which > is suited to static analysis tools, the current proposal (a) confusingly > leaks through into subclass Javadoc and (b) seems too weak. Conclusion: > just dropping @HR from the current scheme (leaving only MHCR <: AC) is an > improvement; we can revisit what the correct annotations for hinting at > resourceful-ness later. > > > > Flaw #2 relates to MHCR. The real problem is that AutoCloseable is > overspecified; it says "a resource that must be closed." This is an > overreach; what j.l.AC should say is "has a close method which is called > automatically on exit from a TWR block." Because of this overspecify, > subclassing doesn't help, because if an AC must be closed, so must a MHCR. > Conclusion: dropping MHCR and just having BaseStream <: AC is an > improvement. > > > > Flaw #3 relates to the spec of AC. The conclusion of the core library > maintainers is that it is acceptable to correct the spec, and replace the > "must" with "may". This leaves us with a fixed AC, and BaseStream <: AC. > This seems strictly better than where we are now. Let's call this "Option > 1." > > > > But, it is possible we might still choose to punt further for now. > > > > Observation: the need for Stream to have a close() method comes about > because of the desire to have static factory methods that will create > streams that encapsulate a resource that requires closing, when there is no > way to get access to that underlying resource. Such as Files.walk(dir) or > Files.lines(path). > > > > For other cases, like BufferedReader.lines(), the BR itself is AC, so > users can handle resource management like: > > > > try (BufferedReader b = ...) { > > b.lines()... > > } > > > > and there is no need to close the *stream* at all, since the user has > access to the actual closeable resource. It is only the static methods > such as Files.list(dir), which don't provide the user access in this way. > The methods we provide are: > > > > Files.list(Path) > > Files.walk(Path, maxDepth, options) > > Files.walk(Path, options) > > Files.find(Path, maxDepth, BiPredicate, > options) > > Files.lines(Path) > > > > Colin @ Google pointed out that if the spliterators for these streams > were lazy (didn't open the resource until the terminal began), and all > terminal ops called the close hook in a finally block after processing, > then we wouldn't need to be AC, since there would be no way to leak a > closeable resource. (This would move resource-not-found exceptions from > the factory method to the terminal op.) > > > > Still, having a close() method and *not* being AC may look weird to the > users. > > > > Let's call this "Option 2": have a close() method, don't extend AC, make > sure that all terminals call close, make sure that all static > resource-holding stream factories produce late-binding spliterators. > > > > Option 3 would be to simply get rid of these methods (note that > Files.list and Files.lines are pretty simple wrappers around existing AC > abstractions, DirectoryStream and BufferedReader.) > > > > Orthogonal to Options 2 and 3 would be fixing the spec of AC. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130803/3f9f8f27/attachment.html From josh at bloch.us Sat Aug 3 09:35:26 2013 From: josh at bloch.us (Joshua Bloch) Date: Sat, 3 Aug 2013 09:35:26 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <253D6820-DDDA-446B-9583-798478F0FD6E@jetbrains.com> Message-ID: That's exactly what I was thinking. The spec for AutoCloseable is not very wrong! I really did intend if for tobjects hings that (almost certainly) need to be closed. ByteArray{In,Out}put Stream are outliers, and (to some extent) historical anomalies. They already extended {In,Out}putStream, which extended Closeable before AutoCloseable was written. And yes, if you took a delegating IO stream (such as Buffered{In,Out}putStream), you could generate another Closeable that didn't really need to be closed. But these were still outliers. To a large extent, Closeable did what it was supposed to. If you have Stream extend Closeable, you're opening the floodgates of Closeables that don't really need to be closed. Josh On Sat, Aug 3, 2013 at 7:54 AM, Joe Bowbeer wrote: > Brian, > > As you wrote, BufferedReader.lines() gets this right and I'm thankful > enough for that: > > try (BufferedReader r = Files.newBufferedReader(path, > Charset.defaultCharset())) { > anagrams(r.lines()).forEach(System.out::println); > } > > There's a similar story with DirectoryStream versus Files.list(), I assume. > > Why aren't there similar formulations for Files.walk and Files.find? > > I'm not convinced that Stream itself should have a close() method; and if > that is troubling to Files.walk, then it can take a hike. > > Fixing the AC spec seems fine, but not relevant to Stream. > > --Joe > > > On Sat, Aug 3, 2013 at 3:10 AM, Andrey Breslav < > andrey.breslav at jetbrains.com> wrote: > >> In case you are asking for opinions, I think >> * we'd better fix the AC spec early than late; >> * have streams implement (fixed) AC (Option 1) looks better to me than >> just have close() (Option 2) or not having the Files.* methods (Option 3). >> >> -- >> Andrey Breslav >> http://jetbrains.com >> Develop with pleasure! >> >> >> On Aug 3, 2013, at 01:39 , Brian Goetz wrote: >> >> > There seems to be consensus that the current strawman for >> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >> > >> > Flaw #1 relates to @HoldsResource. While there seems to be consensus >> that annotations are reasonable for imparting information like this which >> is suited to static analysis tools, the current proposal (a) confusingly >> leaks through into subclass Javadoc and (b) seems too weak. Conclusion: >> just dropping @HR from the current scheme (leaving only MHCR <: AC) is an >> improvement; we can revisit what the correct annotations for hinting at >> resourceful-ness later. >> > >> > Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >> overspecified; it says "a resource that must be closed." This is an >> overreach; what j.l.AC should say is "has a close method which is called >> automatically on exit from a TWR block." Because of this overspecify, >> subclassing doesn't help, because if an AC must be closed, so must a MHCR. >> Conclusion: dropping MHCR and just having BaseStream <: AC is an >> improvement. >> > >> > Flaw #3 relates to the spec of AC. The conclusion of the core library >> maintainers is that it is acceptable to correct the spec, and replace the >> "must" with "may". This leaves us with a fixed AC, and BaseStream <: AC. >> This seems strictly better than where we are now. Let's call this "Option >> 1." >> > >> > But, it is possible we might still choose to punt further for now. >> > >> > Observation: the need for Stream to have a close() method comes about >> because of the desire to have static factory methods that will create >> streams that encapsulate a resource that requires closing, when there is no >> way to get access to that underlying resource. Such as Files.walk(dir) or >> Files.lines(path). >> > >> > For other cases, like BufferedReader.lines(), the BR itself is AC, so >> users can handle resource management like: >> > >> > try (BufferedReader b = ...) { >> > b.lines()... >> > } >> > >> > and there is no need to close the *stream* at all, since the user has >> access to the actual closeable resource. It is only the static methods >> such as Files.list(dir), which don't provide the user access in this way. >> The methods we provide are: >> > >> > Files.list(Path) >> > Files.walk(Path, maxDepth, options) >> > Files.walk(Path, options) >> > Files.find(Path, maxDepth, BiPredicate, >> options) >> > Files.lines(Path) >> > >> > Colin @ Google pointed out that if the spliterators for these streams >> were lazy (didn't open the resource until the terminal began), and all >> terminal ops called the close hook in a finally block after processing, >> then we wouldn't need to be AC, since there would be no way to leak a >> closeable resource. (This would move resource-not-found exceptions from >> the factory method to the terminal op.) >> > >> > Still, having a close() method and *not* being AC may look weird to the >> users. >> > >> > Let's call this "Option 2": have a close() method, don't extend AC, >> make sure that all terminals call close, make sure that all static >> resource-holding stream factories produce late-binding spliterators. >> > >> > Option 3 would be to simply get rid of these methods (note that >> Files.list and Files.lines are pretty simple wrappers around existing AC >> abstractions, DirectoryStream and BufferedReader.) >> > >> > Orthogonal to Options 2 and 3 would be fixing the spec of AC. >> > >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130803/f40f6e05/attachment-0001.html From dl at cs.oswego.edu Sat Aug 3 09:48:45 2013 From: dl at cs.oswego.edu (Doug Lea) Date: Sat, 3 Aug 2013 12:48:45 -0400 (EDT) Subject: Revisiting MayHoldCloseableResource In-Reply-To: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> Message-ID: <52085.67.241.63.52.1375548525.squirrel@altair.cs.oswego.edu> > Observation: the need for Stream to have a close() method comes about > because of the desire to have static factory methods that will create > streams that encapsulate a resource that requires closing, when there is > no way to get access to that underlying resource. Such as Files.walk(dir) > or Files.lines(path). > Or rather, no way provided in JDK. What if these methods were recast do so? That is, exposing some closeable thing. Maybe something like Files.directoryStream(dir).files(). A little less convenient, but avoids lots of problems. > Orthogonal to Options 2 and 3 would be fixing the spec of AC. Definitely. And it is clearly a fix. A spec itself should just say what things do. -Doug From brian.goetz at oracle.com Sat Aug 3 10:14:12 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Sat, 3 Aug 2013 10:14:12 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <52085.67.241.63.52.1375548525.squirrel@altair.cs.oswego.edu> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52085.67.241.63.52.1375548525.squirrel@altair.cs.oswego.edu> Message-ID: >> Observation: the need for Stream to have a close() method comes about >> because of the desire to have static factory methods that will create >> streams that encapsulate a resource that requires closing, when there is >> no way to get access to that underlying resource. Such as Files.walk(dir) >> or Files.lines(path). >> > > Or rather, no way provided in JDK. What if these methods were > recast do so? That is, exposing some closeable thing. > Maybe something like Files.directoryStream(dir).files(). > > A little less convenient, but avoids lots of problems. This was my thinking at first too, but don't forget about cases like flatMap, where the stream is created inside a pipeline. Let's say we do as you suggest for dirStream. But what if you want "all the lines in all those files"? You'd flatMap them to a stream of strings: try (dirStream = ...) { dirStream.stream() .flatMap(path -> Files.lines(path)) ... } There's no place to put TWR around the lines() stream creation even if you expand it as you suggest. This (reasonable!) use case makes me hesitant to solve the problem simply by pretending it didn't occur to us to provide static methods here. From brian.goetz at oracle.com Mon Aug 5 09:39:11 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 5 Aug 2013 09:39:11 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <253D6820-DDDA-446B-9583-798478F0FD6E@jetbrains.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <253D6820-DDDA-446B-9583-798478F0FD6E@jetbrains.com> Message-ID: I'm inclined to agree with Andrey here. The flatMap use case calls for more than just "have a close". On Aug 3, 2013, at 3:10 AM, Andrey Breslav wrote: > In case you are asking for opinions, I think > * we'd better fix the AC spec early than late; > * have streams implement (fixed) AC (Option 1) looks better to me than just have close() (Option 2) or not having the Files.* methods (Option 3). > > -- > Andrey Breslav > http://jetbrains.com > Develop with pleasure! > > > On Aug 3, 2013, at 01:39 , Brian Goetz wrote: > >> There seems to be consensus that the current strawman for MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >> >> Flaw #1 relates to @HoldsResource. While there seems to be consensus that annotations are reasonable for imparting information like this which is suited to static analysis tools, the current proposal (a) confusingly leaks through into subclass Javadoc and (b) seems too weak. Conclusion: just dropping @HR from the current scheme (leaving only MHCR <: AC) is an improvement; we can revisit what the correct annotations for hinting at resourceful-ness later. >> >> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is overspecified; it says "a resource that must be closed." This is an overreach; what j.l.AC should say is "has a close method which is called automatically on exit from a TWR block." Because of this overspecify, subclassing doesn't help, because if an AC must be closed, so must a MHCR. Conclusion: dropping MHCR and just having BaseStream <: AC is an improvement. >> >> Flaw #3 relates to the spec of AC. The conclusion of the core library maintainers is that it is acceptable to correct the spec, and replace the "must" with "may". This leaves us with a fixed AC, and BaseStream <: AC. This seems strictly better than where we are now. Let's call this "Option 1." >> >> But, it is possible we might still choose to punt further for now. >> >> Observation: the need for Stream to have a close() method comes about because of the desire to have static factory methods that will create streams that encapsulate a resource that requires closing, when there is no way to get access to that underlying resource. Such as Files.walk(dir) or Files.lines(path). >> >> For other cases, like BufferedReader.lines(), the BR itself is AC, so users can handle resource management like: >> >> try (BufferedReader b = ...) { >> b.lines()... >> } >> >> and there is no need to close the *stream* at all, since the user has access to the actual closeable resource. It is only the static methods such as Files.list(dir), which don't provide the user access in this way. The methods we provide are: >> >> Files.list(Path) >> Files.walk(Path, maxDepth, options) >> Files.walk(Path, options) >> Files.find(Path, maxDepth, BiPredicate, options) >> Files.lines(Path) >> >> Colin @ Google pointed out that if the spliterators for these streams were lazy (didn't open the resource until the terminal began), and all terminal ops called the close hook in a finally block after processing, then we wouldn't need to be AC, since there would be no way to leak a closeable resource. (This would move resource-not-found exceptions from the factory method to the terminal op.) >> >> Still, having a close() method and *not* being AC may look weird to the users. >> >> Let's call this "Option 2": have a close() method, don't extend AC, make sure that all terminals call close, make sure that all static resource-holding stream factories produce late-binding spliterators. >> >> Option 3 would be to simply get rid of these methods (note that Files.list and Files.lines are pretty simple wrappers around existing AC abstractions, DirectoryStream and BufferedReader.) >> >> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >> >> > From dl at cs.oswego.edu Tue Aug 6 08:43:47 2013 From: dl at cs.oswego.edu (Doug Lea) Date: Tue, 06 Aug 2013 11:43:47 -0400 Subject: FJ parallelism zero In-Reply-To: <5201195D.1050501@cs.oswego.edu> References: <5201195D.1050501@cs.oswego.edu> Message-ID: <520119B3.4000201@cs.oswego.edu> Oops. First posted without the "-libs" On 08/06/13 11:42, Doug Lea wrote: > Mostly to Brian, but other thoughts welcome: > > As a compromise of sorts with the EE folks, we allow the > ForkJoinPool.commonPool to be set to parallelism zero > via a system property. We by default set to > Runtime.availableProcessors()-1. > > All the Stream stuff can handle parallelism zero > (in these cases, the submitter ends up executing > all the subtasks). > > But when the ForkJoinPool.commonPool is used for > purely async stuff, it is surprising at best that > tasks may never run unless somehow joined or the submitter > takes some alternative action (which we sometimes do in > other JDK usages of commonPool). > > I think the EE folks are in general OK with the > surprise. But not so for those few people out > there running JDK8 on uniprocessors, with default > configurations, that also end up setting parallelism > to zero. So unless anyone can think of a reason > otherwise, I'm going to change to minimum parallelism > of one unless set to zero by system property. > (Brian: do you have any recollection of why we didn't > do it this way in the first place?) > > -Doug From brian.goetz at oracle.com Tue Aug 6 12:04:40 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 6 Aug 2013 12:04:40 -0700 Subject: API loose ends Message-ID: Coming into this weeks meeting, we had three API loose ends: - comparing() overloads - collector+finisher combinator - Autocloseable For the first, the EG comments from last week encouraged us to take another look at overloading and see if we can make things yet simpler. Stay tuned for a proposal which would point to an answer on this one. For the second, having explored all the alteranatives, I think the only practical choice is Collectors.comparingAndThen(Collector, Function). For the third, still discussing, will follow up on the appropriate thread. From brian.goetz at oracle.com Fri Aug 9 12:47:16 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 09 Aug 2013 15:47:16 -0400 Subject: Collector / Collectors In-Reply-To: <51DBA05A.8080208@oracle.com> References: <51C0838B.7000703@oracle.com> <51C9F70C.4080801@oracle.com> <51C9FCFB.4050508@oracle.com> <51CB3248.8050102@oracle.com> <51DBA05A.8080208@oracle.com> Message-ID: <52054744.4060605@oracle.com> Having spent a little more time with this and not found a better solution, I think the answer is this: collectingAndThen(coll, fn). Proposed spec (implementation is trivial): /** * Adapts a {@code Collector} to perform an additional finishing * transformation. For example, one could adapt the {@link #toList()} * collector to always produce an immutable list with: * 
{@code
      *     List people
      *         = people.stream().collect(collectingAndThen(toList(), 
Collections::unmodifiableList));
      * }
* * @param the type of the input elements * @param intermediate accumulation type of the downstream collector * @param result type of the downstream collector * @param result type of the resulting collector * @param finisher a function to be applied to the final result of the downstream collector * @param downstream a collector * @return a collector which performs the action of the downstream collector, * followed by an additional finishing step */ public static Collector collectingAndThen(Collector downstream, Function finisher) { On 7/9/2013 1:32 AM, Brian Goetz wrote: > Returning to this... > > We tried the collector.andThen() approach. People liked the API, but we > ran afoul of type inference issues. I think the remaining option is a > combinator: > > collector+finisher -> collector > > but the previously suggested name (finishing) had an "inside out" feel. > But, I think a slight name tweak will do the job: > > groupingBy(Person::getCity, > collectingAndThen(maxBy(comparing(Person::getHeight)), > Optional::get))); > > to suggest what is going on. > > On 6/26/2013 2:26 PM, Brian Goetz wrote: >> >> 2. Making the one-arg reducing(), and minBy/maxBy, return Optional >> means that queries like "tallest person by city" end up with Optional in >> the value: >> >> Map> m >> = people.collect(groupingBy(Person::getCity, >> maxBy(comparing(Person::getHeight))); >> >> Which is doubly bad because the optionals here will *always* be present, >> since otherwise there'd be no associated key. >> >> I can see a few ways to address this: >> - Provide non-optional versions of minBy/maxBy/reducing, which would >> probably have to return null for "no elements". >> - Provide a way to add a finisher to a Collector, which is more >> general (and could be used, say, to turn toList() into something that >> always collects to an immutable list.) >> >> The latter could, in turn, be done in one of two ways: >> >> - Add a andThen(f) method to Collector. Then the above would read: >> >> groupingBy(Person::getCity, >> maxBy(comparing(Person::getHeight)) >> .andThen(Optional::get)) >> >> - Add a combinator: >> >> groupingBy(Person::getCity, >> finishing(maxBy(comparing(Person::getHeight)), >> Optional::get))); >> >> I prefer the former because it reads better in usage; using a combinator >> function feels a little "inside out." From brian.goetz at oracle.com Mon Aug 12 10:03:51 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 12 Aug 2013 13:03:51 -0400 Subject: FJ parallelism zero In-Reply-To: <520119B3.4000201@cs.oswego.edu> References: <5201195D.1050501@cs.oswego.edu> <520119B3.4000201@cs.oswego.edu> Message-ID: <52091577.4050402@oracle.com> No recollection why we didn't do this in the first place! In fact, I don't recall where the -1 came from. The docs for FJP say: * @param parallelism the parallelism level. For default value, * use {@link java.lang.Runtime#availableProcessors}. so I'd assumed that for single-core, we'd set to 1 and it would create a pool with (usually) one thread, and the zero option is "I really want no pool", mostly for those who have to be in control of all thread management (like EE containers.) On 8/6/2013 11:43 AM, Doug Lea wrote: > Oops. First posted without the "-libs" > > On 08/06/13 11:42, Doug Lea wrote: >> Mostly to Brian, but other thoughts welcome: >> >> As a compromise of sorts with the EE folks, we allow the >> ForkJoinPool.commonPool to be set to parallelism zero >> via a system property. We by default set to >> Runtime.availableProcessors()-1. >> >> All the Stream stuff can handle parallelism zero >> (in these cases, the submitter ends up executing >> all the subtasks). >> >> But when the ForkJoinPool.commonPool is used for >> purely async stuff, it is surprising at best that >> tasks may never run unless somehow joined or the submitter >> takes some alternative action (which we sometimes do in >> other JDK usages of commonPool). >> >> I think the EE folks are in general OK with the >> surprise. But not so for those few people out >> there running JDK8 on uniprocessors, with default >> configurations, that also end up setting parallelism >> to zero. So unless anyone can think of a reason >> otherwise, I'm going to change to minimum parallelism >> of one unless set to zero by system property. >> (Brian: do you have any recollection of why we didn't >> do it this way in the first place?) >> >> -Doug > From brian.goetz at oracle.com Mon Aug 12 10:43:43 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 12 Aug 2013 13:43:43 -0400 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> Message-ID: <52091ECF.2060204@oracle.com> After discussion both in the EG and within Oracle, here's where I think this should land: - Eliminate MayHoldCloseableResource, which is fundamentally flawed unless the spec for AutoCloseable is fixed, in which case it becomes unnecessary. - Fix the spec for AutoCloseable. - Eliminate @HoldsResource, not because it is intrinsically bad, but because it is a half solution and we should work towards a complete one for 9, likely using something like JSR-308 checkers. The Eclipse inspector can be adjusted to work with the new APIs. - Make BaseStream implement AutoCloseable, with additional verbiage regarding closing. - Add additional spec to Files.{walk,lines,etc}, and to combinators like concat, regarding their close behavior. The thing that pushed me over the edge from the various "work around the problem" solutions is the existence of flatMap: Files.walk(dir) .flatMap(Files::lines) ... where each entry resulting from walk() will turn into a stream creation and traversal within the flatMap implementation, and flatMap itself needs a way to ensure it is deterministically closed. This pushes strongly for having a close() method on Stream, which, if you're going to have that, you might as well be AutoCloseable to make life easier for users. The sole challenge here is to keep people from thinking that they *need* to close all streams, and therefore mangle their code in bad ways to satisfy this perceived need. On 8/2/2013 7:39 PM, Brian Goetz wrote: > There seems to be consensus that the current strawman for MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. > > Flaw #1 relates to @HoldsResource. While there seems to be consensus that annotations are reasonable for imparting information like this which is suited to static analysis tools, the current proposal (a) confusingly leaks through into subclass Javadoc and (b) seems too weak. Conclusion: just dropping @HR from the current scheme (leaving only MHCR <: AC) is an improvement; we can revisit what the correct annotations for hinting at resourceful-ness later. > > Flaw #2 relates to MHCR. The real problem is that AutoCloseable is overspecified; it says "a resource that must be closed." This is an overreach; what j.l.AC should say is "has a close method which is called automatically on exit from a TWR block." Because of this overspecify, subclassing doesn't help, because if an AC must be closed, so must a MHCR. Conclusion: dropping MHCR and just having BaseStream <: AC is an improvement. > > Flaw #3 relates to the spec of AC. The conclusion of the core library maintainers is that it is acceptable to correct the spec, and replace the "must" with "may". This leaves us with a fixed AC, and BaseStream <: AC. This seems strictly better than where we are now. Let's call this "Option 1." > > But, it is possible we might still choose to punt further for now. > > Observation: the need for Stream to have a close() method comes about because of the desire to have static factory methods that will create streams that encapsulate a resource that requires closing, when there is no way to get access to that underlying resource. Such as Files.walk(dir) or Files.lines(path). > > For other cases, like BufferedReader.lines(), the BR itself is AC, so users can handle resource management like: > > try (BufferedReader b = ...) { > b.lines()... > } > > and there is no need to close the *stream* at all, since the user has access to the actual closeable resource. It is only the static methods such as Files.list(dir), which don't provide the user access in this way. The methods we provide are: > > Files.list(Path) > Files.walk(Path, maxDepth, options) > Files.walk(Path, options) > Files.find(Path, maxDepth, BiPredicate, options) > Files.lines(Path) > > Colin @ Google pointed out that if the spliterators for these streams were lazy (didn't open the resource until the terminal began), and all terminal ops called the close hook in a finally block after processing, then we wouldn't need to be AC, since there would be no way to leak a closeable resource. (This would move resource-not-found exceptions from the factory method to the terminal op.) > > Still, having a close() method and *not* being AC may look weird to the users. > > Let's call this "Option 2": have a close() method, don't extend AC, make sure that all terminals call close, make sure that all static resource-holding stream factories produce late-binding spliterators. > > Option 3 would be to simply get rid of these methods (note that Files.list and Files.lines are pretty simple wrappers around existing AC abstractions, DirectoryStream and BufferedReader.) > > Orthogonal to Options 2 and 3 would be fixing the spec of AC. > > From josh at bloch.us Tue Aug 13 01:03:12 2013 From: josh at bloch.us (Joshua Bloch) Date: Tue, 13 Aug 2013 01:03:12 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <52091ECF.2060204@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> Message-ID: Having BaseStream implement AutoCloseable because some streams need to be closed seems like a bad idea. Josh On Mon, Aug 12, 2013 at 10:43 AM, Brian Goetz wrote: > After discussion both in the EG and within Oracle, here's where I think > this should land: > > - Eliminate MayHoldCloseableResource, which is fundamentally flawed > unless the spec for AutoCloseable is fixed, in which case it becomes > unnecessary. > - Fix the spec for AutoCloseable. > - Eliminate @HoldsResource, not because it is intrinsically bad, but > because it is a half solution and we should work towards a complete one for > 9, likely using something like JSR-308 checkers. The Eclipse inspector can > be adjusted to work with the new APIs. > - Make BaseStream implement AutoCloseable, with additional verbiage > regarding closing. > - Add additional spec to Files.{walk,lines,etc}, and to combinators like > concat, regarding their close behavior. > > The thing that pushed me over the edge from the various "work around the > problem" solutions is the existence of flatMap: > > Files.walk(dir) > .flatMap(Files::lines) > ... > > where each entry resulting from walk() will turn into a stream creation > and traversal within the flatMap implementation, and flatMap itself needs a > way to ensure it is deterministically closed. This pushes strongly for > having a close() method on Stream, which, if you're going to have that, you > might as well be AutoCloseable to make life easier for users. > > The sole challenge here is to keep people from thinking that they *need* > to close all streams, and therefore mangle their code in bad ways to > satisfy this perceived need. > > > > On 8/2/2013 7:39 PM, Brian Goetz wrote: > >> There seems to be consensus that the current strawman for >> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >> >> Flaw #1 relates to @HoldsResource. While there seems to be consensus >> that annotations are reasonable for imparting information like this which >> is suited to static analysis tools, the current proposal (a) confusingly >> leaks through into subclass Javadoc and (b) seems too weak. Conclusion: >> just dropping @HR from the current scheme (leaving only MHCR <: AC) is an >> improvement; we can revisit what the correct annotations for hinting at >> resourceful-ness later. >> >> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >> overspecified; it says "a resource that must be closed." This is an >> overreach; what j.l.AC should say is "has a close method which is called >> automatically on exit from a TWR block." Because of this overspecify, >> subclassing doesn't help, because if an AC must be closed, so must a MHCR. >> Conclusion: dropping MHCR and just having BaseStream <: AC is an >> improvement. >> >> Flaw #3 relates to the spec of AC. The conclusion of the core library >> maintainers is that it is acceptable to correct the spec, and replace the >> "must" with "may". This leaves us with a fixed AC, and BaseStream <: AC. >> This seems strictly better than where we are now. Let's call this "Option >> 1." >> >> But, it is possible we might still choose to punt further for now. >> >> Observation: the need for Stream to have a close() method comes about >> because of the desire to have static factory methods that will create >> streams that encapsulate a resource that requires closing, when there is no >> way to get access to that underlying resource. Such as Files.walk(dir) or >> Files.lines(path). >> >> For other cases, like BufferedReader.lines(), the BR itself is AC, so >> users can handle resource management like: >> >> try (BufferedReader b = ...) { >> b.lines()... >> } >> >> and there is no need to close the *stream* at all, since the user has >> access to the actual closeable resource. It is only the static methods >> such as Files.list(dir), which don't provide the user access in this way. >> The methods we provide are: >> >> Files.list(Path) >> Files.walk(Path, maxDepth, options) >> Files.walk(Path, options) >> Files.find(Path, maxDepth, BiPredicate, >> options) >> Files.lines(Path) >> >> Colin @ Google pointed out that if the spliterators for these streams >> were lazy (didn't open the resource until the terminal began), and all >> terminal ops called the close hook in a finally block after processing, >> then we wouldn't need to be AC, since there would be no way to leak a >> closeable resource. (This would move resource-not-found exceptions from >> the factory method to the terminal op.) >> >> Still, having a close() method and *not* being AC may look weird to the >> users. >> >> Let's call this "Option 2": have a close() method, don't extend AC, make >> sure that all terminals call close, make sure that all static >> resource-holding stream factories produce late-binding spliterators. >> >> Option 3 would be to simply get rid of these methods (note that >> Files.list and Files.lines are pretty simple wrappers around existing AC >> abstractions, DirectoryStream and BufferedReader.) >> >> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130813/ad385aaa/attachment.html From brian.goetz at oracle.com Tue Aug 13 10:03:47 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 13 Aug 2013 13:03:47 -0400 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <52091ECF.2060204@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> Message-ID: <520A66F3.1020603@oracle.com> OK, here's proposed spec: For AC: /** - * A resource that must be closed when it is no longer needed. + * A resource that may need to be closed when it is no longer needed, and whose + * {@link #close()} method will be called automatically when exiting a + * {@code try}-with-resources block for which the resource has been declared in + * the header. A resource would need to be explicitly closed if it holds a + * scarce system resource that is not promptly reclaimed by garbage collection, + * such as a file or socket handle. + * + * @apiNote + *

It should generally be possible to determine statically whether a resource + * that may need to be closed actually does need to be closed (in which case one + * should consider using try-with-resources to ensure its timely closure). + * For example, most {@link InputStream} implementations need to be closed, + * though {@link ByteArrayInputStream} does not need to be; most {@link Stream} + * implementations do not need to be closed, but streams which encapsulate + * IO resources, like those produced by the static factories in {@link Files} + * such as {@link Files#walk(Path, FileVisitOption...)}, do need to be closed. For BaseStream and friends: + *

Streams have a {@link #close()} method and implement {@link AutoCloseable}, + * but nearly all stream instances do not actually need to be closed after use. + * Generally, only streams whose source is an IO channel (such as those returned + * by {@link Files#lines(Path, Charset)}) will require closing. Most streams + * are backed by collections, arrays, or generating functions, which require no + * special resource management. (If a stream does require closing, it can be + * declared as a resource in a {@code try}-with-resources statement.) + * For Files.walk and friends: + *

The returned stream encapsulates one or more {@link DirectoryStream}s. + * If timely disposal of file system resources is required, the + * {@code try}-with-resources construct should be used to ensure that the + * stream's {@link Stream#close close} method is invoked after the stream + * operations are completed. On 8/12/2013 1:43 PM, Brian Goetz wrote: > After discussion both in the EG and within Oracle, here's where I think > this should land: > > - Eliminate MayHoldCloseableResource, which is fundamentally flawed > unless the spec for AutoCloseable is fixed, in which case it becomes > unnecessary. > - Fix the spec for AutoCloseable. > - Eliminate @HoldsResource, not because it is intrinsically bad, but > because it is a half solution and we should work towards a complete one > for 9, likely using something like JSR-308 checkers. The Eclipse > inspector can be adjusted to work with the new APIs. > - Make BaseStream implement AutoCloseable, with additional verbiage > regarding closing. > - Add additional spec to Files.{walk,lines,etc}, and to combinators > like concat, regarding their close behavior. > > The thing that pushed me over the edge from the various "work around the > problem" solutions is the existence of flatMap: > > Files.walk(dir) > .flatMap(Files::lines) > ... > > where each entry resulting from walk() will turn into a stream creation > and traversal within the flatMap implementation, and flatMap itself > needs a way to ensure it is deterministically closed. This pushes > strongly for having a close() method on Stream, which, if you're going > to have that, you might as well be AutoCloseable to make life easier for > users. > > The sole challenge here is to keep people from thinking that they *need* > to close all streams, and therefore mangle their code in bad ways to > satisfy this perceived need. > > > On 8/2/2013 7:39 PM, Brian Goetz wrote: >> There seems to be consensus that the current strawman for >> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >> >> Flaw #1 relates to @HoldsResource. While there seems to be consensus >> that annotations are reasonable for imparting information like this >> which is suited to static analysis tools, the current proposal (a) >> confusingly leaks through into subclass Javadoc and (b) seems too >> weak. Conclusion: just dropping @HR from the current scheme (leaving >> only MHCR <: AC) is an improvement; we can revisit what the correct >> annotations for hinting at resourceful-ness later. >> >> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >> overspecified; it says "a resource that must be closed." This is an >> overreach; what j.l.AC should say is "has a close method which is >> called automatically on exit from a TWR block." Because of this >> overspecify, subclassing doesn't help, because if an AC must be >> closed, so must a MHCR. Conclusion: dropping MHCR and just having >> BaseStream <: AC is an improvement. >> >> Flaw #3 relates to the spec of AC. The conclusion of the core library >> maintainers is that it is acceptable to correct the spec, and replace >> the "must" with "may". This leaves us with a fixed AC, and BaseStream >> <: AC. This seems strictly better than where we are now. Let's call >> this "Option 1." >> >> But, it is possible we might still choose to punt further for now. >> >> Observation: the need for Stream to have a close() method comes about >> because of the desire to have static factory methods that will create >> streams that encapsulate a resource that requires closing, when there >> is no way to get access to that underlying resource. Such as >> Files.walk(dir) or Files.lines(path). >> >> For other cases, like BufferedReader.lines(), the BR itself is AC, so >> users can handle resource management like: >> >> try (BufferedReader b = ...) { >> b.lines()... >> } >> >> and there is no need to close the *stream* at all, since the user has >> access to the actual closeable resource. It is only the static >> methods such as Files.list(dir), which don't provide the user access >> in this way. The methods we provide are: >> >> Files.list(Path) >> Files.walk(Path, maxDepth, options) >> Files.walk(Path, options) >> Files.find(Path, maxDepth, BiPredicate, >> options) >> Files.lines(Path) >> >> Colin @ Google pointed out that if the spliterators for these streams >> were lazy (didn't open the resource until the terminal began), and all >> terminal ops called the close hook in a finally block after >> processing, then we wouldn't need to be AC, since there would be no >> way to leak a closeable resource. (This would move resource-not-found >> exceptions from the factory method to the terminal op.) >> >> Still, having a close() method and *not* being AC may look weird to >> the users. >> >> Let's call this "Option 2": have a close() method, don't extend AC, >> make sure that all terminals call close, make sure that all static >> resource-holding stream factories produce late-binding spliterators. >> >> Option 3 would be to simply get rid of these methods (note that >> Files.list and Files.lines are pretty simple wrappers around existing >> AC abstractions, DirectoryStream and BufferedReader.) >> >> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >> >> From kevinb at google.com Tue Aug 13 10:24:58 2013 From: kevinb at google.com (Kevin Bourrillion) Date: Tue, 13 Aug 2013 10:24:58 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> Message-ID: On Tue, Aug 13, 2013 at 1:03 AM, Joshua Bloch wrote: Having BaseStream implement AutoCloseable because some streams need to be > closed seems like a bad idea. > Yet, alternatives were heavily discussed and this looks like the viable solution. We have to view AutoCloseable as exposing the *ability* to be closed (which may be a no-op), not as implying a burden to the caller that it *must* be closed. ByteArrayInputStream etc. already show that we can't effectively surface that responsibility in the type hierarchy alone. There needs to be a separate means for static analyses to judge whether an instance *needed* to be closed. > > Josh > > > On Mon, Aug 12, 2013 at 10:43 AM, Brian Goetz wrote: > >> After discussion both in the EG and within Oracle, here's where I think >> this should land: >> >> - Eliminate MayHoldCloseableResource, which is fundamentally flawed >> unless the spec for AutoCloseable is fixed, in which case it becomes >> unnecessary. >> - Fix the spec for AutoCloseable. >> - Eliminate @HoldsResource, not because it is intrinsically bad, but >> because it is a half solution and we should work towards a complete one for >> 9, likely using something like JSR-308 checkers. The Eclipse inspector can >> be adjusted to work with the new APIs. >> - Make BaseStream implement AutoCloseable, with additional verbiage >> regarding closing. >> - Add additional spec to Files.{walk,lines,etc}, and to combinators like >> concat, regarding their close behavior. >> >> The thing that pushed me over the edge from the various "work around the >> problem" solutions is the existence of flatMap: >> >> Files.walk(dir) >> .flatMap(Files::lines) >> ... >> >> where each entry resulting from walk() will turn into a stream creation >> and traversal within the flatMap implementation, and flatMap itself needs a >> way to ensure it is deterministically closed. This pushes strongly for >> having a close() method on Stream, which, if you're going to have that, you >> might as well be AutoCloseable to make life easier for users. >> >> The sole challenge here is to keep people from thinking that they *need* >> to close all streams, and therefore mangle their code in bad ways to >> satisfy this perceived need. >> >> >> >> On 8/2/2013 7:39 PM, Brian Goetz wrote: >> >>> There seems to be consensus that the current strawman for >>> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >>> >>> Flaw #1 relates to @HoldsResource. While there seems to be consensus >>> that annotations are reasonable for imparting information like this which >>> is suited to static analysis tools, the current proposal (a) confusingly >>> leaks through into subclass Javadoc and (b) seems too weak. Conclusion: >>> just dropping @HR from the current scheme (leaving only MHCR <: AC) is an >>> improvement; we can revisit what the correct annotations for hinting at >>> resourceful-ness later. >>> >>> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >>> overspecified; it says "a resource that must be closed." This is an >>> overreach; what j.l.AC should say is "has a close method which is >>> called automatically on exit from a TWR block." Because of this >>> overspecify, subclassing doesn't help, because if an AC must be closed, so >>> must a MHCR. Conclusion: dropping MHCR and just having BaseStream <: AC is >>> an improvement. >>> >>> Flaw #3 relates to the spec of AC. The conclusion of the core library >>> maintainers is that it is acceptable to correct the spec, and replace the >>> "must" with "may". This leaves us with a fixed AC, and BaseStream <: AC. >>> This seems strictly better than where we are now. Let's call this "Option >>> 1." >>> >>> But, it is possible we might still choose to punt further for now. >>> >>> Observation: the need for Stream to have a close() method comes about >>> because of the desire to have static factory methods that will create >>> streams that encapsulate a resource that requires closing, when there is no >>> way to get access to that underlying resource. Such as Files.walk(dir) or >>> Files.lines(path). >>> >>> For other cases, like BufferedReader.lines(), the BR itself is AC, so >>> users can handle resource management like: >>> >>> try (BufferedReader b = ...) { >>> b.lines()... >>> } >>> >>> and there is no need to close the *stream* at all, since the user has >>> access to the actual closeable resource. It is only the static methods >>> such as Files.list(dir), which don't provide the user access in this way. >>> The methods we provide are: >>> >>> Files.list(Path) >>> Files.walk(Path, maxDepth, options) >>> Files.walk(Path, options) >>> Files.find(Path, maxDepth, BiPredicate, >>> options) >>> Files.lines(Path) >>> >>> Colin @ Google pointed out that if the spliterators for these streams >>> were lazy (didn't open the resource until the terminal began), and all >>> terminal ops called the close hook in a finally block after processing, >>> then we wouldn't need to be AC, since there would be no way to leak a >>> closeable resource. (This would move resource-not-found exceptions from >>> the factory method to the terminal op.) >>> >>> Still, having a close() method and *not* being AC may look weird to the >>> users. >>> >>> Let's call this "Option 2": have a close() method, don't extend AC, make >>> sure that all terminals call close, make sure that all static >>> resource-holding stream factories produce late-binding spliterators. >>> >>> Option 3 would be to simply get rid of these methods (note that >>> Files.list and Files.lines are pretty simple wrappers around existing AC >>> abstractions, DirectoryStream and BufferedReader.) >>> >>> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >>> >>> >>> > -- Kevin Bourrillion | Java Librarian | Google, Inc. | kevinb at google.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130813/81b47c37/attachment-0001.html From spullara at gmail.com Tue Aug 13 10:43:08 2013 From: spullara at gmail.com (Sam Pullara) Date: Tue, 13 Aug 2013 10:43:08 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <520A66F3.1020603@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> Message-ID: Isn't Closeable basically defined this way? "A Closeable is a source or destination of data that can be closed. The close method is invoked to release resources that the object is holding (such as open files)." It is interesting to me that Closeable extends AutoCloseable and the latter has a stricter requirements in the Javadocs. Sam On Aug 13, 2013, at 10:03 AM, Brian Goetz wrote: > OK, here's proposed spec: > > For AC: > > /** > - * A resource that must be closed when it is no longer needed. > + * A resource that may need to be closed when it is no longer needed, and whose > + * {@link #close()} method will be called automatically when exiting a > + * {@code try}-with-resources block for which the resource has been declared in > + * the header. A resource would need to be explicitly closed if it holds a > + * scarce system resource that is not promptly reclaimed by garbage collection, > + * such as a file or socket handle. > + * > + * @apiNote > + *

It should generally be possible to determine statically whether a resource > + * that may need to be closed actually does need to be closed (in which case one > + * should consider using try-with-resources to ensure its timely closure). > + * For example, most {@link InputStream} implementations need to be closed, > + * though {@link ByteArrayInputStream} does not need to be; most {@link Stream} > + * implementations do not need to be closed, but streams which encapsulate > + * IO resources, like those produced by the static factories in {@link Files} > + * such as {@link Files#walk(Path, FileVisitOption...)}, do need to be closed. > > For BaseStream and friends: > > + *

Streams have a {@link #close()} method and implement {@link AutoCloseable}, > + * but nearly all stream instances do not actually need to be closed after use. > + * Generally, only streams whose source is an IO channel (such as those returned > + * by {@link Files#lines(Path, Charset)}) will require closing. Most streams > + * are backed by collections, arrays, or generating functions, which require no > + * special resource management. (If a stream does require closing, it can be > + * declared as a resource in a {@code try}-with-resources statement.) > + * > > For Files.walk and friends: > > + *

The returned stream encapsulates one or more {@link DirectoryStream}s. > + * If timely disposal of file system resources is required, the > + * {@code try}-with-resources construct should be used to ensure that the > + * stream's {@link Stream#close close} method is invoked after the stream > + * operations are completed. > > > On 8/12/2013 1:43 PM, Brian Goetz wrote: >> After discussion both in the EG and within Oracle, here's where I think >> this should land: >> >> - Eliminate MayHoldCloseableResource, which is fundamentally flawed >> unless the spec for AutoCloseable is fixed, in which case it becomes >> unnecessary. >> - Fix the spec for AutoCloseable. >> - Eliminate @HoldsResource, not because it is intrinsically bad, but >> because it is a half solution and we should work towards a complete one >> for 9, likely using something like JSR-308 checkers. The Eclipse >> inspector can be adjusted to work with the new APIs. >> - Make BaseStream implement AutoCloseable, with additional verbiage >> regarding closing. >> - Add additional spec to Files.{walk,lines,etc}, and to combinators >> like concat, regarding their close behavior. >> >> The thing that pushed me over the edge from the various "work around the >> problem" solutions is the existence of flatMap: >> >> Files.walk(dir) >> .flatMap(Files::lines) >> ... >> >> where each entry resulting from walk() will turn into a stream creation >> and traversal within the flatMap implementation, and flatMap itself >> needs a way to ensure it is deterministically closed. This pushes >> strongly for having a close() method on Stream, which, if you're going >> to have that, you might as well be AutoCloseable to make life easier for >> users. >> >> The sole challenge here is to keep people from thinking that they *need* >> to close all streams, and therefore mangle their code in bad ways to >> satisfy this perceived need. >> >> >> On 8/2/2013 7:39 PM, Brian Goetz wrote: >>> There seems to be consensus that the current strawman for >>> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >>> >>> Flaw #1 relates to @HoldsResource. While there seems to be consensus >>> that annotations are reasonable for imparting information like this >>> which is suited to static analysis tools, the current proposal (a) >>> confusingly leaks through into subclass Javadoc and (b) seems too >>> weak. Conclusion: just dropping @HR from the current scheme (leaving >>> only MHCR <: AC) is an improvement; we can revisit what the correct >>> annotations for hinting at resourceful-ness later. >>> >>> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >>> overspecified; it says "a resource that must be closed." This is an >>> overreach; what j.l.AC should say is "has a close method which is >>> called automatically on exit from a TWR block." Because of this >>> overspecify, subclassing doesn't help, because if an AC must be >>> closed, so must a MHCR. Conclusion: dropping MHCR and just having >>> BaseStream <: AC is an improvement. >>> >>> Flaw #3 relates to the spec of AC. The conclusion of the core library >>> maintainers is that it is acceptable to correct the spec, and replace >>> the "must" with "may". This leaves us with a fixed AC, and BaseStream >>> <: AC. This seems strictly better than where we are now. Let's call >>> this "Option 1." >>> >>> But, it is possible we might still choose to punt further for now. >>> >>> Observation: the need for Stream to have a close() method comes about >>> because of the desire to have static factory methods that will create >>> streams that encapsulate a resource that requires closing, when there >>> is no way to get access to that underlying resource. Such as >>> Files.walk(dir) or Files.lines(path). >>> >>> For other cases, like BufferedReader.lines(), the BR itself is AC, so >>> users can handle resource management like: >>> >>> try (BufferedReader b = ...) { >>> b.lines()... >>> } >>> >>> and there is no need to close the *stream* at all, since the user has >>> access to the actual closeable resource. It is only the static >>> methods such as Files.list(dir), which don't provide the user access >>> in this way. The methods we provide are: >>> >>> Files.list(Path) >>> Files.walk(Path, maxDepth, options) >>> Files.walk(Path, options) >>> Files.find(Path, maxDepth, BiPredicate, >>> options) >>> Files.lines(Path) >>> >>> Colin @ Google pointed out that if the spliterators for these streams >>> were lazy (didn't open the resource until the terminal began), and all >>> terminal ops called the close hook in a finally block after >>> processing, then we wouldn't need to be AC, since there would be no >>> way to leak a closeable resource. (This would move resource-not-found >>> exceptions from the factory method to the terminal op.) >>> >>> Still, having a close() method and *not* being AC may look weird to >>> the users. >>> >>> Let's call this "Option 2": have a close() method, don't extend AC, >>> make sure that all terminals call close, make sure that all static >>> resource-holding stream factories produce late-binding spliterators. >>> >>> Option 3 would be to simply get rid of these methods (note that >>> Files.list and Files.lines are pretty simple wrappers around existing >>> AC abstractions, DirectoryStream and BufferedReader.) >>> >>> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130813/2d7b711d/attachment.html From brian.goetz at oracle.com Tue Aug 13 10:45:44 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 13 Aug 2013 13:45:44 -0400 Subject: Revisiting MayHoldCloseableResource In-Reply-To: References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> Message-ID: <520A70C8.3050105@oracle.com> I think the Closeable <: AutoCloseable relation is consistent with the observations about the over-specification of AC in Java 7. An AC *can* be closed, and works with TWR. A Closeable is a java.io thing that *should* be closed (but again here, even ByteArrayInputStream is doc'ed to say that close() is a no-op.) So the spec of Closeable is probably also too strong (though not suggesting we fix that now.) On 8/13/2013 1:43 PM, Sam Pullara wrote: > Isn't Closeable basically defined this way? > > "A|Closeable|is a source or destination of data that can be closed. The > close method is invoked to release resources that the object is holding > (such as open files)." > > It is interesting to me that Closeable extends AutoCloseable and the > latter has a stricter requirements in the Javadocs. > > Sam > > On Aug 13, 2013, at 10:03 AM, Brian Goetz > wrote: > >> OK, here's proposed spec: >> >> For AC: >> >> /** >> - * A resource that must be closed when it is no longer needed. >> + * A resource that may need to be closed when it is no longer needed, >> and whose >> + * {@link #close()} method will be called automatically when exiting a >> + * {@code try}-with-resources block for which the resource has been >> declared in >> + * the header. A resource would need to be explicitly closed if it >> holds a >> + * scarce system resource that is not promptly reclaimed by garbage >> collection, >> + * such as a file or socket handle. >> + * >> + * @apiNote >> + *

It should generally be possible to determine statically whether >> a resource >> + * that may need to be closed actually does need to be closed (in >> which case one >> + * should consider using try-with-resources to ensure its timely >> closure). >> + * For example, most {@link InputStream} implementations need to be >> closed, >> + * though {@link ByteArrayInputStream} does not need to be; most >> {@link Stream} >> + * implementations do not need to be closed, but streams >> which encapsulate >> + * IO resources, like those produced by the static factories in >> {@link Files} >> + * such as {@link Files#walk(Path, FileVisitOption...)}, do need to >> be closed. >> >> For BaseStream and friends: >> >> + *

Streams have a {@link #close()} method and implement {@link >> AutoCloseable}, >> + * but nearly all stream instances do not actually need to be closed >> after use. >> + * Generally, only streams whose source is an IO channel (such as >> those returned >> + * by {@link Files#lines(Path, Charset)}) will require closing. Most >> streams >> + * are backed by collections, arrays, or generating functions, which >> require no >> + * special resource management. (If a stream does require closing, >> it can be >> + * declared as a resource in a {@code try}-with-resources statement.) >> + * >> >> For Files.walk and friends: >> >> + *

The returned stream encapsulates one or more {@link >> DirectoryStream}s. >> + * If timely disposal of file system resources is required, the >> + * {@code try}-with-resources construct should be used to ensure >> that the >> + * stream's {@link Stream#close close} method is invoked after >> the stream >> + * operations are completed. >> >> >> On 8/12/2013 1:43 PM, Brian Goetz wrote: >>> After discussion both in the EG and within Oracle, here's where I think >>> this should land: >>> >>> - Eliminate MayHoldCloseableResource, which is fundamentally flawed >>> unless the spec for AutoCloseable is fixed, in which case it becomes >>> unnecessary. >>> - Fix the spec for AutoCloseable. >>> - Eliminate @HoldsResource, not because it is intrinsically bad, but >>> because it is a half solution and we should work towards a complete one >>> for 9, likely using something like JSR-308 checkers. The Eclipse >>> inspector can be adjusted to work with the new APIs. >>> - Make BaseStream implement AutoCloseable, with additional verbiage >>> regarding closing. >>> - Add additional spec to Files.{walk,lines,etc}, and to combinators >>> like concat, regarding their close behavior. >>> >>> The thing that pushed me over the edge from the various "work around the >>> problem" solutions is the existence of flatMap: >>> >>> Files.walk(dir) >>> .flatMap(Files::lines) >>> ... >>> >>> where each entry resulting from walk() will turn into a stream creation >>> and traversal within the flatMap implementation, and flatMap itself >>> needs a way to ensure it is deterministically closed. This pushes >>> strongly for having a close() method on Stream, which, if you're going >>> to have that, you might as well be AutoCloseable to make life easier for >>> users. >>> >>> The sole challenge here is to keep people from thinking that they *need* >>> to close all streams, and therefore mangle their code in bad ways to >>> satisfy this perceived need. >>> >>> >>> On 8/2/2013 7:39 PM, Brian Goetz wrote: >>>> There seems to be consensus that the current strawman for >>>> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >>>> >>>> Flaw #1 relates to @HoldsResource. While there seems to be consensus >>>> that annotations are reasonable for imparting information like this >>>> which is suited to static analysis tools, the current proposal (a) >>>> confusingly leaks through into subclass Javadoc and (b) seems too >>>> weak. Conclusion: just dropping @HR from the current scheme (leaving >>>> only MHCR <: AC) is an improvement; we can revisit what the correct >>>> annotations for hinting at resourceful-ness later. >>>> >>>> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >>>> overspecified; it says "a resource that must be closed." This is an >>>> overreach; what j.l.AC should say is "has a close method which is >>>> called automatically on exit from a TWR block." Because of this >>>> overspecify, subclassing doesn't help, because if an AC must be >>>> closed, so must a MHCR. Conclusion: dropping MHCR and just having >>>> BaseStream <: AC is an improvement. >>>> >>>> Flaw #3 relates to the spec of AC. The conclusion of the core library >>>> maintainers is that it is acceptable to correct the spec, and replace >>>> the "must" with "may". This leaves us with a fixed AC, and BaseStream >>>> <: AC. This seems strictly better than where we are now. Let's call >>>> this "Option 1." >>>> >>>> But, it is possible we might still choose to punt further for now. >>>> >>>> Observation: the need for Stream to have a close() method comes about >>>> because of the desire to have static factory methods that will create >>>> streams that encapsulate a resource that requires closing, when there >>>> is no way to get access to that underlying resource. Such as >>>> Files.walk(dir) or Files.lines(path). >>>> >>>> For other cases, like BufferedReader.lines(), the BR itself is AC, so >>>> users can handle resource management like: >>>> >>>> try (BufferedReader b = ...) { >>>> b.lines()... >>>> } >>>> >>>> and there is no need to close the *stream* at all, since the user has >>>> access to the actual closeable resource. It is only the static >>>> methods such as Files.list(dir), which don't provide the user access >>>> in this way. The methods we provide are: >>>> >>>> Files.list(Path) >>>> Files.walk(Path, maxDepth, options) >>>> Files.walk(Path, options) >>>> Files.find(Path, maxDepth, BiPredicate, >>>> options) >>>> Files.lines(Path) >>>> >>>> Colin @ Google pointed out that if the spliterators for these streams >>>> were lazy (didn't open the resource until the terminal began), and all >>>> terminal ops called the close hook in a finally block after >>>> processing, then we wouldn't need to be AC, since there would be no >>>> way to leak a closeable resource. (This would move resource-not-found >>>> exceptions from the factory method to the terminal op.) >>>> >>>> Still, having a close() method and *not* being AC may look weird to >>>> the users. >>>> >>>> Let's call this "Option 2": have a close() method, don't extend AC, >>>> make sure that all terminals call close, make sure that all static >>>> resource-holding stream factories produce late-binding spliterators. >>>> >>>> Option 3 would be to simply get rid of these methods (note that >>>> Files.list and Files.lines are pretty simple wrappers around existing >>>> AC abstractions, DirectoryStream and BufferedReader.) >>>> >>>> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >>>> >>>> > From joe.bowbeer at gmail.com Tue Aug 13 10:47:03 2013 From: joe.bowbeer at gmail.com (Joe Bowbeer) Date: Tue, 13 Aug 2013 10:47:03 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> Message-ID: Another unfortunate thing is the name close() which is, I think, a little stricter than we want here. dispose() or release() would have been better -- because they convey an intent not to use anymore, but do not imply that a synchronous transition to the "closed" state should be visible to every observer immediately. But, alas, neither dispose() nor release() are viable alternatives at this time. On Tue, Aug 13, 2013 at 10:24 AM, Kevin Bourrillion wrote: > On Tue, Aug 13, 2013 at 1:03 AM, Joshua Bloch wrote: > > Having BaseStream implement AutoCloseable because some streams need to be >> closed seems like a bad idea. >> > > Yet, alternatives were heavily discussed and this looks like the viable > solution. > > We have to view AutoCloseable as exposing the *ability* to be closed > (which may be a no-op), not as implying a burden to the caller that it * > must* be closed. ByteArrayInputStream etc. already show that we can't > effectively surface that responsibility in the type hierarchy alone. There > needs to be a separate means for static analyses to judge whether an > instance *needed* to be closed. > > > >> >> Josh >> >> >> On Mon, Aug 12, 2013 at 10:43 AM, Brian Goetz wrote: >> >>> After discussion both in the EG and within Oracle, here's where I think >>> this should land: >>> >>> - Eliminate MayHoldCloseableResource, which is fundamentally flawed >>> unless the spec for AutoCloseable is fixed, in which case it becomes >>> unnecessary. >>> - Fix the spec for AutoCloseable. >>> - Eliminate @HoldsResource, not because it is intrinsically bad, but >>> because it is a half solution and we should work towards a complete one for >>> 9, likely using something like JSR-308 checkers. The Eclipse inspector can >>> be adjusted to work with the new APIs. >>> - Make BaseStream implement AutoCloseable, with additional verbiage >>> regarding closing. >>> - Add additional spec to Files.{walk,lines,etc}, and to combinators >>> like concat, regarding their close behavior. >>> >>> The thing that pushed me over the edge from the various "work around the >>> problem" solutions is the existence of flatMap: >>> >>> Files.walk(dir) >>> .flatMap(Files::lines) >>> ... >>> >>> where each entry resulting from walk() will turn into a stream creation >>> and traversal within the flatMap implementation, and flatMap itself needs a >>> way to ensure it is deterministically closed. This pushes strongly for >>> having a close() method on Stream, which, if you're going to have that, you >>> might as well be AutoCloseable to make life easier for users. >>> >>> The sole challenge here is to keep people from thinking that they *need* >>> to close all streams, and therefore mangle their code in bad ways to >>> satisfy this perceived need. >>> >>> >>> >>> On 8/2/2013 7:39 PM, Brian Goetz wrote: >>> >>>> There seems to be consensus that the current strawman for >>>> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >>>> >>>> Flaw #1 relates to @HoldsResource. While there seems to be consensus >>>> that annotations are reasonable for imparting information like this which >>>> is suited to static analysis tools, the current proposal (a) confusingly >>>> leaks through into subclass Javadoc and (b) seems too weak. Conclusion: >>>> just dropping @HR from the current scheme (leaving only MHCR <: AC) is an >>>> improvement; we can revisit what the correct annotations for hinting at >>>> resourceful-ness later. >>>> >>>> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >>>> overspecified; it says "a resource that must be closed." This is an >>>> overreach; what j.l.AC should say is "has a close method which is >>>> called automatically on exit from a TWR block." Because of this >>>> overspecify, subclassing doesn't help, because if an AC must be closed, so >>>> must a MHCR. Conclusion: dropping MHCR and just having BaseStream <: AC is >>>> an improvement. >>>> >>>> Flaw #3 relates to the spec of AC. The conclusion of the core library >>>> maintainers is that it is acceptable to correct the spec, and replace the >>>> "must" with "may". This leaves us with a fixed AC, and BaseStream <: AC. >>>> This seems strictly better than where we are now. Let's call this "Option >>>> 1." >>>> >>>> But, it is possible we might still choose to punt further for now. >>>> >>>> Observation: the need for Stream to have a close() method comes about >>>> because of the desire to have static factory methods that will create >>>> streams that encapsulate a resource that requires closing, when there is no >>>> way to get access to that underlying resource. Such as Files.walk(dir) or >>>> Files.lines(path). >>>> >>>> For other cases, like BufferedReader.lines(), the BR itself is AC, so >>>> users can handle resource management like: >>>> >>>> try (BufferedReader b = ...) { >>>> b.lines()... >>>> } >>>> >>>> and there is no need to close the *stream* at all, since the user has >>>> access to the actual closeable resource. It is only the static methods >>>> such as Files.list(dir), which don't provide the user access in this way. >>>> The methods we provide are: >>>> >>>> Files.list(Path) >>>> Files.walk(Path, maxDepth, options) >>>> Files.walk(Path, options) >>>> Files.find(Path, maxDepth, BiPredicate, >>>> options) >>>> Files.lines(Path) >>>> >>>> Colin @ Google pointed out that if the spliterators for these streams >>>> were lazy (didn't open the resource until the terminal began), and all >>>> terminal ops called the close hook in a finally block after processing, >>>> then we wouldn't need to be AC, since there would be no way to leak a >>>> closeable resource. (This would move resource-not-found exceptions from >>>> the factory method to the terminal op.) >>>> >>>> Still, having a close() method and *not* being AC may look weird to the >>>> users. >>>> >>>> Let's call this "Option 2": have a close() method, don't extend AC, >>>> make sure that all terminals call close, make sure that all static >>>> resource-holding stream factories produce late-binding spliterators. >>>> >>>> Option 3 would be to simply get rid of these methods (note that >>>> Files.list and Files.lines are pretty simple wrappers around existing AC >>>> abstractions, DirectoryStream and BufferedReader.) >>>> >>>> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >>>> >>>> >>>> >> > > > -- > Kevin Bourrillion | Java Librarian | Google, Inc. | kevinb at google.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130813/318e2a2d/attachment.html From brian.goetz at oracle.com Tue Aug 13 11:13:38 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 13 Aug 2013 14:13:38 -0400 Subject: Reader mail bag, libraries edition Message-ID: <520A7752.6090407@oracle.com> As you may know, there is a separate -comments list which is intended to be used as a "suggestion box" (i.e., its not for discussion, and hence not subscribable or replyable.) This is the public's primary means for providing input to the EG. Most have already been discussed here. The following comments should be considered "read into the record". If folks want to discuss these, please start a separate thread. Dec 10, 2012, Paul Benedict: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-comments/2012-December/000000.html Summary: Collapse stream() and parallelStream() into stream(Hint) Disposition: Reasonable idea, but too early to standardize the form of the API (symptom: everyone has a different suggestion for how this should look), will reconsider for 9 Jan 24, 2013, Michael Hixson: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-comments/2013-January/000001.html Summary: More convenience methods for Comparator Disposition: partially incorporated into API Jan 31, 2013, Michael Hixson: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-comments/2013-January/000004.html Summary: Narrow return type of Comparator.thenComparing Disposition: Prototyped and rejected due to interactions with type inference Mar 27, 2013, David Illsley: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-comments/2013-March/000005.html Summary: Please add Predicates.notNull Disposition: Not added; x -> x != null is no more verbose and no less abstract From brian.goetz at oracle.com Tue Aug 13 11:34:32 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 13 Aug 2013 14:34:32 -0400 Subject: From the issue tracker Message-ID: <520A7C38.8010600@oracle.com> As of today, the following issues are still open on the JSR-335 issue tracker hosted at java.net: https://java.net/jira/browse/JSR_335-5 Summary: What shoudl the specified toString behavior of lambdas be? Disposition: no decision reached https://java.net/jira/browse/JSR_335-12 Summary: Concerns about serialization stability Disposition: Issue re-raised on EG list If anyone has (new) observations about either of these topics, please start a separate thread. From brian.goetz at oracle.com Tue Aug 13 12:05:05 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 13 Aug 2013 15:05:05 -0400 Subject: Spec review request In-Reply-To: <51DEED9C.2050106@oracle.com> References: <51DC6E21.2050405@oracle.com> <51DEED9C.2050106@oracle.com> Message-ID: <520A8361.2000707@oracle.com> NO reviews received to date. Please, help! On 7/11/2013 1:38 PM, Brian Goetz wrote: > I've updated the package doc as well. So the doc set can now be > considered complete and ready for review. > > On 7/9/2013 4:10 PM, Brian Goetz wrote: >> I believe the spec for all classes in java.util.stream is ready for API >> review. Please see current docs here: >> >> http://cr.openjdk.java.net/~briangoetz/doctmp/doc/ >> >> Some notes: >> - The package docs are not yet done; when they are finished, links to >> the package docs will be added in appropriate places. >> - Ignore instances of ... in the doc. This is an >> artifact of @link'ing to classes in a package for which doc is not >> generated. >> - Any loose ends still in flight may have yet to be reflected. >> From dl at cs.oswego.edu Wed Aug 14 06:09:53 2013 From: dl at cs.oswego.edu (Doug Lea) Date: Wed, 14 Aug 2013 09:09:53 -0400 Subject: FJ parallelism zero In-Reply-To: <52091577.4050402@oracle.com> References: <5201195D.1050501@cs.oswego.edu> <520119B3.4000201@cs.oswego.edu> <52091577.4050402@oracle.com> Message-ID: <520B81A1.2090203@cs.oswego.edu> On 08/12/13 13:03, Brian Goetz wrote: > No recollection why we didn't do this in the first place! In fact, I don't > recall where the -1 came from. The docs for FJP say: > > * @param parallelism the parallelism level. For default value, > * use {@link java.lang.Runtime#availableProcessors}. Not that it matters at this point since we are committing the change, but the "-1" was done separately last fall to compensate for the caller-runs changes in FJP. Without it, you can get more threads working at any given time than cores -- the workers, plus the caller. This is generally fine but suboptimal. But should have still been floored at one to cover asyncs. -Doug > > so I'd assumed that for single-core, we'd set to 1 and it would create a pool > with (usually) one thread, and the zero option is "I really want no pool", > mostly for those who have to be in control of all thread management (like EE > containers.) > > On 8/6/2013 11:43 AM, Doug Lea wrote: >> Oops. First posted without the "-libs" >> >> On 08/06/13 11:42, Doug Lea wrote: >>> Mostly to Brian, but other thoughts welcome: >>> >>> As a compromise of sorts with the EE folks, we allow the >>> ForkJoinPool.commonPool to be set to parallelism zero >>> via a system property. We by default set to >>> Runtime.availableProcessors()-1. >>> >>> All the Stream stuff can handle parallelism zero >>> (in these cases, the submitter ends up executing >>> all the subtasks). >>> >>> But when the ForkJoinPool.commonPool is used for >>> purely async stuff, it is surprising at best that >>> tasks may never run unless somehow joined or the submitter >>> takes some alternative action (which we sometimes do in >>> other JDK usages of commonPool). >>> >>> I think the EE folks are in general OK with the >>> surprise. But not so for those few people out >>> there running JDK8 on uniprocessors, with default >>> configurations, that also end up setting parallelism >>> to zero. So unless anyone can think of a reason >>> otherwise, I'm going to change to minimum parallelism >>> of one unless set to zero by system property. >>> (Brian: do you have any recollection of why we didn't >>> do it this way in the first place?) >>> >>> -Doug >> > From brian.goetz at oracle.com Wed Aug 14 12:49:45 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 14 Aug 2013 15:49:45 -0400 Subject: Comparator combinators Message-ID: <520BDF59.3080100@oracle.com> This may well be our last API loose end... We currently have a pile of Comparator combinators, all currently called comparing() or thenComparing(). Regardless of whether we choose to go forward with the simplified overloading plan, these overloads have a problem: for implicit lambdas, we can't distinguish between comparing(T->U) and comparing(T->int) because we don't type-check the body of the lambda until we do overload selection, and don't do overload selection based on whether there are type errors in the body (this was roundly rejected as being too brittle). So for lambdas like: comparing(x -> x.size()) we've got a circularity -- even under the older, more complex scheme. We'd thought perhaps that, if we adopted the heuristic that we can eagerly give a type to method reference that refers to a non-overloaded method (e.g., Person::getAge), then cases like comparing(Person::getAge) can be handled, and this might take some of the pressure off the problem. For lambdas (with either approach) you can always get what you want with an explicit lambda: comparing((Person p) -> p.getAge()) since this can be type-checked early. So the question is, is this good enough, even though it falls afoul of the overloading guidelines for implicit lambdas? Or, should we mangle the names of the methods? This question comes down to whether we think its better to force everyone to explicitly pick a method, but then supporting implicit lambdas: comparingInt(x -> x.size()) or forcing users to use non-ambigous method refs or explicit lambdas: comparing(Person::getAge) or comparing((Person p) -> p.getAge()) Which disambiguation approach is worse? From brian.goetz at oracle.com Thu Aug 15 13:25:54 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 15 Aug 2013 16:25:54 -0400 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <520A66F3.1020603@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> Message-ID: <520D3952.30208@oracle.com> Last chance for comment. On 8/13/2013 1:03 PM, Brian Goetz wrote: > OK, here's proposed spec: > > For AC: > > /** > - * A resource that must be closed when it is no longer needed. > + * A resource that may need to be closed when it is no longer needed, > and whose > + * {@link #close()} method will be called automatically when exiting a > + * {@code try}-with-resources block for which the resource has been > declared in > + * the header. A resource would need to be explicitly closed if it > holds a > + * scarce system resource that is not promptly reclaimed by garbage > collection, > + * such as a file or socket handle. > + * > + * @apiNote > + *

It should generally be possible to determine statically whether a > resource > + * that may need to be closed actually does need to be closed (in which > case one > + * should consider using try-with-resources to ensure its timely closure). > + * For example, most {@link InputStream} implementations need to be > closed, > + * though {@link ByteArrayInputStream} does not need to be; most {@link > Stream} > + * implementations do not need to be closed, but streams which > encapsulate > + * IO resources, like those produced by the static factories in {@link > Files} > + * such as {@link Files#walk(Path, FileVisitOption...)}, do need to be > closed. > > For BaseStream and friends: > > + *

Streams have a {@link #close()} method and implement {@link > AutoCloseable}, > + * but nearly all stream instances do not actually need to be closed > after use. > + * Generally, only streams whose source is an IO channel (such as those > returned > + * by {@link Files#lines(Path, Charset)}) will require closing. Most > streams > + * are backed by collections, arrays, or generating functions, which > require no > + * special resource management. (If a stream does require closing, it > can be > + * declared as a resource in a {@code try}-with-resources statement.) > + * > > For Files.walk and friends: > > + *

The returned stream encapsulates one or more {@link > DirectoryStream}s. > + * If timely disposal of file system resources is required, the > + * {@code try}-with-resources construct should be used to ensure > that the > + * stream's {@link Stream#close close} method is invoked after the > stream > + * operations are completed. > > > On 8/12/2013 1:43 PM, Brian Goetz wrote: >> After discussion both in the EG and within Oracle, here's where I think >> this should land: >> >> - Eliminate MayHoldCloseableResource, which is fundamentally flawed >> unless the spec for AutoCloseable is fixed, in which case it becomes >> unnecessary. >> - Fix the spec for AutoCloseable. >> - Eliminate @HoldsResource, not because it is intrinsically bad, but >> because it is a half solution and we should work towards a complete one >> for 9, likely using something like JSR-308 checkers. The Eclipse >> inspector can be adjusted to work with the new APIs. >> - Make BaseStream implement AutoCloseable, with additional verbiage >> regarding closing. >> - Add additional spec to Files.{walk,lines,etc}, and to combinators >> like concat, regarding their close behavior. >> >> The thing that pushed me over the edge from the various "work around the >> problem" solutions is the existence of flatMap: >> >> Files.walk(dir) >> .flatMap(Files::lines) >> ... >> >> where each entry resulting from walk() will turn into a stream creation >> and traversal within the flatMap implementation, and flatMap itself >> needs a way to ensure it is deterministically closed. This pushes >> strongly for having a close() method on Stream, which, if you're going >> to have that, you might as well be AutoCloseable to make life easier for >> users. >> >> The sole challenge here is to keep people from thinking that they *need* >> to close all streams, and therefore mangle their code in bad ways to >> satisfy this perceived need. >> >> >> On 8/2/2013 7:39 PM, Brian Goetz wrote: >>> There seems to be consensus that the current strawman for >>> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >>> >>> Flaw #1 relates to @HoldsResource. While there seems to be consensus >>> that annotations are reasonable for imparting information like this >>> which is suited to static analysis tools, the current proposal (a) >>> confusingly leaks through into subclass Javadoc and (b) seems too >>> weak. Conclusion: just dropping @HR from the current scheme (leaving >>> only MHCR <: AC) is an improvement; we can revisit what the correct >>> annotations for hinting at resourceful-ness later. >>> >>> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >>> overspecified; it says "a resource that must be closed." This is an >>> overreach; what j.l.AC should say is "has a close method which is >>> called automatically on exit from a TWR block." Because of this >>> overspecify, subclassing doesn't help, because if an AC must be >>> closed, so must a MHCR. Conclusion: dropping MHCR and just having >>> BaseStream <: AC is an improvement. >>> >>> Flaw #3 relates to the spec of AC. The conclusion of the core library >>> maintainers is that it is acceptable to correct the spec, and replace >>> the "must" with "may". This leaves us with a fixed AC, and BaseStream >>> <: AC. This seems strictly better than where we are now. Let's call >>> this "Option 1." >>> >>> But, it is possible we might still choose to punt further for now. >>> >>> Observation: the need for Stream to have a close() method comes about >>> because of the desire to have static factory methods that will create >>> streams that encapsulate a resource that requires closing, when there >>> is no way to get access to that underlying resource. Such as >>> Files.walk(dir) or Files.lines(path). >>> >>> For other cases, like BufferedReader.lines(), the BR itself is AC, so >>> users can handle resource management like: >>> >>> try (BufferedReader b = ...) { >>> b.lines()... >>> } >>> >>> and there is no need to close the *stream* at all, since the user has >>> access to the actual closeable resource. It is only the static >>> methods such as Files.list(dir), which don't provide the user access >>> in this way. The methods we provide are: >>> >>> Files.list(Path) >>> Files.walk(Path, maxDepth, options) >>> Files.walk(Path, options) >>> Files.find(Path, maxDepth, BiPredicate, >>> options) >>> Files.lines(Path) >>> >>> Colin @ Google pointed out that if the spliterators for these streams >>> were lazy (didn't open the resource until the terminal began), and all >>> terminal ops called the close hook in a finally block after >>> processing, then we wouldn't need to be AC, since there would be no >>> way to leak a closeable resource. (This would move resource-not-found >>> exceptions from the factory method to the terminal op.) >>> >>> Still, having a close() method and *not* being AC may look weird to >>> the users. >>> >>> Let's call this "Option 2": have a close() method, don't extend AC, >>> make sure that all terminals call close, make sure that all static >>> resource-holding stream factories produce late-binding spliterators. >>> >>> Option 3 would be to simply get rid of these methods (note that >>> Files.list and Files.lines are pretty simple wrappers around existing >>> AC abstractions, DirectoryStream and BufferedReader.) >>> >>> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >>> >>> From josh at bloch.us Thu Aug 15 16:43:27 2013 From: josh at bloch.us (Joshua Bloch) Date: Thu, 15 Aug 2013 16:43:27 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <520D3952.30208@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> <520D3952.30208@oracle.com> Message-ID: You know what I think. I wouldn't do what you're going to do. Josh On Thu, Aug 15, 2013 at 1:25 PM, Brian Goetz wrote: > Last chance for comment. > > > On 8/13/2013 1:03 PM, Brian Goetz wrote: > >> OK, here's proposed spec: >> >> For AC: >> >> /** >> - * A resource that must be closed when it is no longer needed. >> + * A resource that may need to be closed when it is no longer needed, >> and whose >> + * {@link #close()} method will be called automatically when exiting a >> + * {@code try}-with-resources block for which the resource has been >> declared in >> + * the header. A resource would need to be explicitly closed if it >> holds a >> + * scarce system resource that is not promptly reclaimed by garbage >> collection, >> + * such as a file or socket handle. >> + * >> + * @apiNote >> + *

It should generally be possible to determine statically whether a >> resource >> + * that may need to be closed actually does need to be closed (in which >> case one >> + * should consider using try-with-resources to ensure its timely >> closure). >> + * For example, most {@link InputStream} implementations need to be >> closed, >> + * though {@link ByteArrayInputStream} does not need to be; most {@link >> Stream} >> + * implementations do not need to be closed, but streams which >> encapsulate >> + * IO resources, like those produced by the static factories in {@link >> Files} >> + * such as {@link Files#walk(Path, FileVisitOption...)}, do need to be >> closed. >> >> For BaseStream and friends: >> >> + *

Streams have a {@link #close()} method and implement {@link >> AutoCloseable}, >> + * but nearly all stream instances do not actually need to be closed >> after use. >> + * Generally, only streams whose source is an IO channel (such as those >> returned >> + * by {@link Files#lines(Path, Charset)}) will require closing. Most >> streams >> + * are backed by collections, arrays, or generating functions, which >> require no >> + * special resource management. (If a stream does require closing, it >> can be >> + * declared as a resource in a {@code try}-with-resources statement.) >> + * >> >> For Files.walk and friends: >> >> + *

The returned stream encapsulates one or more {@link >> DirectoryStream}s. >> + * If timely disposal of file system resources is required, the >> + * {@code try}-with-resources construct should be used to ensure >> that the >> + * stream's {@link Stream#close close} method is invoked after the >> stream >> + * operations are completed. >> >> >> On 8/12/2013 1:43 PM, Brian Goetz wrote: >> >>> After discussion both in the EG and within Oracle, here's where I think >>> this should land: >>> >>> - Eliminate MayHoldCloseableResource, which is fundamentally flawed >>> unless the spec for AutoCloseable is fixed, in which case it becomes >>> unnecessary. >>> - Fix the spec for AutoCloseable. >>> - Eliminate @HoldsResource, not because it is intrinsically bad, but >>> because it is a half solution and we should work towards a complete one >>> for 9, likely using something like JSR-308 checkers. The Eclipse >>> inspector can be adjusted to work with the new APIs. >>> - Make BaseStream implement AutoCloseable, with additional verbiage >>> regarding closing. >>> - Add additional spec to Files.{walk,lines,etc}, and to combinators >>> like concat, regarding their close behavior. >>> >>> The thing that pushed me over the edge from the various "work around the >>> problem" solutions is the existence of flatMap: >>> >>> Files.walk(dir) >>> .flatMap(Files::lines) >>> ... >>> >>> where each entry resulting from walk() will turn into a stream creation >>> and traversal within the flatMap implementation, and flatMap itself >>> needs a way to ensure it is deterministically closed. This pushes >>> strongly for having a close() method on Stream, which, if you're going >>> to have that, you might as well be AutoCloseable to make life easier for >>> users. >>> >>> The sole challenge here is to keep people from thinking that they *need* >>> to close all streams, and therefore mangle their code in bad ways to >>> satisfy this perceived need. >>> >>> >>> On 8/2/2013 7:39 PM, Brian Goetz wrote: >>> >>>> There seems to be consensus that the current strawman for >>>> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >>>> >>>> Flaw #1 relates to @HoldsResource. While there seems to be consensus >>>> that annotations are reasonable for imparting information like this >>>> which is suited to static analysis tools, the current proposal (a) >>>> confusingly leaks through into subclass Javadoc and (b) seems too >>>> weak. Conclusion: just dropping @HR from the current scheme (leaving >>>> only MHCR <: AC) is an improvement; we can revisit what the correct >>>> annotations for hinting at resourceful-ness later. >>>> >>>> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >>>> overspecified; it says "a resource that must be closed." This is an >>>> overreach; what j.l.AC should say is "has a close method which is >>>> called automatically on exit from a TWR block." Because of this >>>> overspecify, subclassing doesn't help, because if an AC must be >>>> closed, so must a MHCR. Conclusion: dropping MHCR and just having >>>> BaseStream <: AC is an improvement. >>>> >>>> Flaw #3 relates to the spec of AC. The conclusion of the core library >>>> maintainers is that it is acceptable to correct the spec, and replace >>>> the "must" with "may". This leaves us with a fixed AC, and BaseStream >>>> <: AC. This seems strictly better than where we are now. Let's call >>>> this "Option 1." >>>> >>>> But, it is possible we might still choose to punt further for now. >>>> >>>> Observation: the need for Stream to have a close() method comes about >>>> because of the desire to have static factory methods that will create >>>> streams that encapsulate a resource that requires closing, when there >>>> is no way to get access to that underlying resource. Such as >>>> Files.walk(dir) or Files.lines(path). >>>> >>>> For other cases, like BufferedReader.lines(), the BR itself is AC, so >>>> users can handle resource management like: >>>> >>>> try (BufferedReader b = ...) { >>>> b.lines()... >>>> } >>>> >>>> and there is no need to close the *stream* at all, since the user has >>>> access to the actual closeable resource. It is only the static >>>> methods such as Files.list(dir), which don't provide the user access >>>> in this way. The methods we provide are: >>>> >>>> Files.list(Path) >>>> Files.walk(Path, maxDepth, options) >>>> Files.walk(Path, options) >>>> Files.find(Path, maxDepth, BiPredicate, >>>> options) >>>> Files.lines(Path) >>>> >>>> Colin @ Google pointed out that if the spliterators for these streams >>>> were lazy (didn't open the resource until the terminal began), and all >>>> terminal ops called the close hook in a finally block after >>>> processing, then we wouldn't need to be AC, since there would be no >>>> way to leak a closeable resource. (This would move resource-not-found >>>> exceptions from the factory method to the terminal op.) >>>> >>>> Still, having a close() method and *not* being AC may look weird to >>>> the users. >>>> >>>> Let's call this "Option 2": have a close() method, don't extend AC, >>>> make sure that all terminals call close, make sure that all static >>>> resource-holding stream factories produce late-binding spliterators. >>>> >>>> Option 3 would be to simply get rid of these methods (note that >>>> Files.list and Files.lines are pretty simple wrappers around existing >>>> AC abstractions, DirectoryStream and BufferedReader.) >>>> >>>> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >>>> >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130815/c4334088/attachment.html From henry.jen at oracle.com Thu Aug 15 17:38:23 2013 From: henry.jen at oracle.com (Henry Jen) Date: Thu, 15 Aug 2013 17:38:23 -0700 Subject: Comparator combinators In-Reply-To: <520BDF59.3080100@oracle.com> References: <520BDF59.3080100@oracle.com> Message-ID: <7CA6D7B4-7400-498E-A328-5F54D8F337FC@oracle.com> I prefer leave it as is and force use of explicit lambda or non-ambiguous method reference. For one reason John had mentioned before, when one intend to comparingInt(x -> x.size()) and wrote comparing(x -> x.size()) instead, boxing kicks in and the code still works. Forcing to write comparing((IntFunction) x -> x.size()) is not too bad IMHO. My feeling is that most commonly use cases are covered in test/java/util/Comparator/BasicTest.java, and those compiles fine without extra casting. Cheers, Henry On Aug 14, 2013, at 12:49 PM, Brian Goetz wrote: > This may well be our last API loose end... > > We currently have a pile of Comparator combinators, all currently called comparing() or thenComparing(). Regardless of whether we choose to go forward with the simplified overloading plan, these overloads have a problem: for implicit lambdas, we can't distinguish between > > comparing(T->U) > and > comparing(T->int) > > because we don't type-check the body of the lambda until we do overload selection, and don't do overload selection based on whether there are type errors in the body (this was roundly rejected as being too brittle). So for lambdas like: > > comparing(x -> x.size()) > > we've got a circularity -- even under the older, more complex scheme. > > We'd thought perhaps that, if we adopted the heuristic that we can eagerly give a type to method reference that refers to a non-overloaded method (e.g., Person::getAge), then cases like > > comparing(Person::getAge) > > can be handled, and this might take some of the pressure off the problem. > > For lambdas (with either approach) you can always get what you want with an explicit lambda: > > comparing((Person p) -> p.getAge()) > > since this can be type-checked early. > > So the question is, is this good enough, even though it falls afoul of the overloading guidelines for implicit lambdas? Or, should we mangle the names of the methods? > > This question comes down to whether we think its better to force everyone to explicitly pick a method, but then supporting implicit lambdas: > > comparingInt(x -> x.size()) > > or forcing users to use non-ambigous method refs or explicit lambdas: > > comparing(Person::getAge) > or > comparing((Person p) -> p.getAge()) > > Which disambiguation approach is worse? > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130815/b368ee03/attachment-0001.html From brian.goetz at oracle.com Thu Aug 15 18:04:26 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 15 Aug 2013 21:04:26 -0400 Subject: Comparator combinators In-Reply-To: <7CA6D7B4-7400-498E-A328-5F54D8F337FC@oracle.com> References: <520BDF59.3080100@oracle.com> <7CA6D7B4-7400-498E-A328-5F54D8F337FC@oracle.com> Message-ID: <520D7A9A.3000202@oracle.com> > comparing((IntFunction) x -> x.size()) is not too bad IMHO. or comparing((Foo x) -> x.size()) Its not completely intuitive that adding the argument type helps the compiler determine the best return type, but it does, because it enables the lambda to be type-checked bottom up (since it contains no inference variables anymore.) From brian.goetz at oracle.com Fri Aug 16 10:47:52 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 16 Aug 2013 13:47:52 -0400 Subject: Serializable lambdas -- where we are, how we got here Message-ID: <520E65C8.4010600@oracle.com> Several concerns have been recently (re)raised again about the stability of serializable lambdas. This attempts to provide an inventory of where we are and how we got here. There were some who initially (wishfully) suggested that it would be best to declare serialization a mistake and not make lambdas serializable at all. While this was a very tempting target, ultimately this conflicted with another decision we made: that of using nominal function types (functional interfaces) to type lambdas. For example, imagine: interface SerializablePredicate extends Predicate, Serializable { } If the user does: SerializablePredicate p = s -> false; or SerializablePredicate p = String::isEmpty; It would violate the principle of least surprise that the resulting objects (whose lambda-heritage should be invisible to anyone who later touches it) to not be serializable. Hence begun our slide down the slippery slope. An intrinsic challenge of serialization is that, when confronted with different class files at deserialization time than were present at serialization time, to make a good-faith effort to figure out what to do. For classes, the default behavior (in the absence of an explicit serial version UID) is to consider any change to the class signatures to invalidate existing serialized forms, but in the presence of a serial version UID, to attempt to deal gracefully with added or removed fields. Inherent in this is the assumption that if the *name* and *signature* of something hasn't changed, its semantics haven't, either. If you change the meaning of a field or a method, but not its name, you're out of luck. Anonymous classes are less forgiving than nominal classes, because (a) their names are generated at compile time and may change if the source changes "too much", and (b) their field names / constructor signature may change based on changes in method bodies even if the class and method signatures don't change. This problem has been with us since 1997. There are two possible failure modes that come out of this: Type 1) An instance may fail to deserialize, due to changes that have nothing to do with the object being serialized; Type 2) An instance may deserialize successfully, but may be bound to the *wrong* implementation due to bad luck. Still, many users successfully deal with serialization and anonymous classes by following a simple rule: have the same bits on both sides of the wire. In reality, the situation is more forgiving than that: if you recompile the same source with the same compiler, things still work -- and users fundamentally expect this to be the case. And the same is true for "lightly modified" versions of the same sources (adding comments, adding debugging statements, etc.) Lambdas are similar to anonymous classes in some ways, and we were aware of these failure modes at the time we first discussed serialization of lambdas. Obviously we would have preferred to prevent these failures if possible, but all the approaches explored were either too restrictive or incomplete. Restrictions that were explored and rejected include: - No serializable lambdas at all - Only serialize static or unbound method refs - Only serialize named, non-capturing lambdas The various hash-the-world options that have been suggested (hash the source or bytecode) are too weird, too brittle, too hard to specify, and will result in users being confounded by, say, recompiling what they perceive as identical sources with an identical compiler and still getting runtime failures, violating (reasonable) user expectations. (It would be almost better to generate a *random* name on every compilation, but we're not going to do that.) In the absence of being able to make it perfect, having exactly the same drawbacks of an existing mechanism, which users are familiar with and have learned to work around, was deemed better than making it imperfect in yet a new way. That said, if there's a possibility to reduce type-2 failures without undermining the usability of serialization or the simplicity of the user model, we're willing to continue to explore these (despite the extreme lateness of the hour). At the recent EG meeting, we specifically discussed whether it would be worthwhile to try and address recovering from capture-order issues. This *is* tractible (subject to the same caveats with nominal classes -- that same-name means same-meaning). But, the sense of the room then was that this doesn't help enough, because there is still the name-induced stability issue, and that fixing one without the other just encourages users to think that they can make arbitrary code changes and expect serialization stability, and makes it even more surprising when we get a failure due to, say, adding a new lambda to a method. However, if we felt we were likely to do named lambdas later, then this approach could close half the problem now and we could close the other half of the problem later. One possibility that has not yet been discussed is to issue a lint warning for serializable lambdas/method refs that are subject to stability issues. Here's where we are: - We're not revisiting the decisions about what lambdas and method references should be serializable. This has been reopened several times with no change in consensus, and no new information has come to light that would change the decision. - "Just like inner classes" is a local maxima. Better to not ask the user to create a new mental model than to require a new one that is just as flawed but in different ways. However, we already make some departures from inner class treatment, so this is a more "spirit of the rule" thing than a "letter of the rule." If we can do *much* better, great, but "slightly better but different" is worse. - We might be able to revisit some translation decisions if they result in significant improvements to stability without cost to usability, but we are almost, if not completely, out of time. - We're open to adding more lint warnings at compile time. Stay tuned for a specific proposal. From brian.goetz at oracle.com Mon Aug 19 08:37:08 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 19 Aug 2013 11:37:08 -0400 Subject: One final stab at improving lambda serialization Message-ID: <52123BA4.4020700@oracle.com> *Background* The fundamental challenge with serialization is that the code that defined a class at serialization time may have changed by the time deserialization happens. Serialization is defined to be tolerant of change to a certain extent, and admits a degree of customization to allow additional flexibility. For ordinary classes, there are three lines of defense: * serialVersionUID * serialization hooks * default schema evolution Serial version UID for the target class must match exactly. By default, serialization uses a serial version UID which is a hash of the classes signatures. So this default approach means "any significant change to the structure of the class (adding new methods, changing method or field signatures, etc) renders serialized forms invalid". It is a common practice to explicitly assign a serial version UID to a class, thereby disabling this mechanism. Classes that expect to evolve over time may use readObject/writeObject and/or readResolve/writeReplace to customize the mapping between object state and bytestream. If classes do not use this mechanism, serialization uses a default schema evolution mechanism to adjust for changes in fields between serialization and deserialization time; fields that are present in the bytestream but not in the target class are ignored, and fields that are present in the target class but not the bytestream get default values (zero, null, etc.) Anonymous classes follow the same approach and have access to the same mechanisms (serialVersionUID, read/writeObject, etc), but they have two additional sources of instability: * The name is generated as EnclosingClass$nnn. Any change to the set of anonymous classes in the enclosing class may cause sequence numbers to change. * The number and type of fields (appears in bytecode but not source code) are generated based on the set of captured values. Any change to the set or order of captured values can cause these signatures to change (in an unspecified way). If the signatures remain stable, anonymous classes can use serialization hooks to customize the serialized form, just like named classes. The EG has observed that users have largely learned to deal with the problems of serialization of inner classes, either by (a) don't do it, or (b) ensure that essentially the same bits are present on both sides of the pipe, preventing skew from causing instability in either class names or signatures. The EG has set, as a minimum bar, that lambda serialization be "at least as good as" anonymous class serialization. (This is not a high bar.) Further, the EG has concluded that gratuitous deviations from anonymous class serialization are undesirable, because, if users have to deal with an imperfect scheme, having them deal with something that is basically the same as an imperfect scheme they've already gotten used to is preferable to dealing with a new and different scheme. Further, the EG has rejected the idea of arbitrarily restricting access to serialization just because it is dangerous; users who have learned to use it safely should not be unduly encumbered. *Failure modes * For anonymous classes, one of two things will happen when attempting to deserialize after things have changed "too much": 1. A deserialization failure due to either the name or signature not matching, resulting in NoSuchMethodError, IncompatibleClassChangeError, etc. 2. Deserializing to the wrong thing, without any evidence of error. Obviously, a type-2 failure is far worse than a type-1 failure, because no error is raised and an unintended computation is performed. Here are two examples of changes that are behaviorally compatible but which will result in type-2 failures. The first has to do with order-of-declaration. *Old code** * *New code** * *Result** * Runnable r1 = new Runnable() { void run() { System.out.println("one"); } }; Runnable r2 = new Runnable() { void run() { System.out.println("two"); } }; Runnable r2 = new Runnable() { void run() { System.out.println("two"); } }; Runnable r1 = new Runnable() { void run() { System.out.println("one"); } }; Deserialized r1 (across skew) prints "two". This fails because in both cases, we get classes called Foo$1 and Foo$2, but in the old code, these correspond to r1 and r2, but in the new code, these correspond to r2 and r1. The other failure has to do with order-of-capture. *Old code** * *New code** * *Result** * String s1 = "foo"; String s2 = "bar"; Runnable r = new Runnable() { void run() { foo(s1, s2); } }; String s1 = "foo"; String s2 = "bar"; Runnable r = new Runnable() { void run() { String s = s2; foo(s1, s); } }; On deserialization, s1 and s2 are effectively swapped. This fails because the order of arguments in the implicitly generated constructor of the inner class changes due to the order in which the compiler encounters captured variables. If the reordered variables were of different types, this would cause a type-1 failure, but if they are the same type, it causes a type-2 failure. *User expectations* While experienced users are quick to state the "same bits on both sides" rule for reliable deserialization, a bit of investigation reveals that user expectations are actually higher than that. For example, if the compiler generated a /random/ name for each lambda at compile time, then recompiling the same source with the same compiler, and using the result for deserialization, would fail. This is too restrictive; user expectations are not tied to "same bits", but to a vaguer notion of "I compiled essentially the same source with essentially the same compiler, and therefore didn't change anything significant." For example, users would balk if adding a comment or changing whitespace were to affect deserialization. Users likely expect (in part, due to behavior of anonymous classes) changes to code that doesn't affect the lambda directly or indirectly (e.g., add or remove a debugging println) also would not affect the serialized form. In the absence of the user being able to explicitly name the lambda /and/ its captures (as C++ does), there is no perfect solution. Instead, our goal can only be to minimize type-2 failures while not unduly creating type-1 failures when "no significant code change" happened. This means we have to put a stake in the ground as to what constitutes "significant" code change. The de-facto (and likely accidental) definition of "significant" used by inner classes here is: * Adding, removing, or reordering inner class instances earlier in the source file; * Changes to the number, order, or type of captured arguments This permits changes to code that has nothing to do with inner classes, and many common refactorings as long as they do not affect the order of inner class instances or their captures. *Current Lambda behavior* Lambda serialization currently behaves very similarly to anonymous class serialization. Where anonymous classes have stable method names but unstable class names, lambdas are the dual; unstable method names but stable class names. But since both are used together, the resulting naming stability is largely the same. We do one thing to increase naming stability for lambdas: we hash the name and signature of the enclosing method in the lambda name. This insulates lambda naming from the addition, removal, or reordering of methods within a class file, but naming stability remains sensitive to the order of lambdas within the method. Similarly, order-of-capture issues are largely similar to inner classes. Lambdas bodies are desugared to methods named in the following form: lambda$/mmm/$/nnn/, where /mmm/ is a hash of the method name and signature, and /nnn/ is a sequence number of lambdas that have the same /mmm/ hash. Because lambdas are instantiated via invokedynamic rather than invoking a constructor directly, there is also slightly more leniency to changes to the /types/ of captured argument; changing a captured argument from, say, String to Object, would be a breaking change for anonymous classes (it changes the constructor signature) but not for lambdas. This leniency is largely an accidental artifact of translation, rather than a deliberate design decision. *Possible improvements* We can start by recognizing the role of the hash of the enclosing method in the lambda method name. This reduces the set of lambdas that could collide from "all the lambdas in the file" to "all the lambdas in the method." This reduces the set of changes that cause both type-1 and type-2 errors. An additional observation is that there is a tension between trying to /recover from/ skew (rather than simply trying to detect it, and failing deserialization) and complexity. So I think we should focus primarily on detecting skew and failing deserialization (turning type-2 failures into type-1) while at the same time not unduly increasing the set of changes that cause type-1 errors, with the goal of settling on an informal guideline of what constitutes "too much" change. We can do this by increasing the number of things that affect the /mmm/ hash, effectively constructing the lambda-equivalent of the serialization version UID. The more context we add to this hash, the smaller the set of lambdas that hash to the same bucket gets, which reduces the space of possible collisions. The following table shows possible candidates for inclusion, along with examples of code that illustrate dependence on this item. *Item** * *Old Code** ------------------------------ * *New Code** **------------------------------* *Effect** * *Rationale** * Names of captured arguments int x = ... f(() -> x); int y = ... f(() -> y); Including the names of captured arguments in the hash would cause rename-refactors of captured arguments to be considered a serialization-breaking change. While alpha-renaming is generally considered to be semantic-preserving, serialization has always keyed off of names (such as field names) as being clues to developer intent. It seems reasonable to say "If you change the names involved, we have to assume a semantic change occurred." We cannot tell if a name change is a simple alpha-rename or capturing a completely different variable, so this is erring on the safe side. Types of captured arguments String x = ... f(() -> x); Object x = ... f(() -> x); It seems reasonable to say that, if you capture arguments of a different type, you've made a semantic change. Order of captured arguments () -> { int a = f(x); int b = g(y); return h(a,b); }; () -> { int b = g(y); int a = f(x); return h(a,b); }; Changing the order of capture would become a type-1 failure rather than possibly a type-2 failure. Since we cannot detect whether the ordering change is semantically meaningful or not, it is best to be conservative and say: change to capture order is likely a semantic change. Variable assignment target (if present) Runnable r1 = Foo::f; Runnable r2 = Foo::g; Runnable r2 = Foo::g; Runnable r1 = Foo::f; Including variable target name would render this reordering recoverable and correct If the user has gone to the effort of providing a name, we can use this as a hint to the meaning of the lambda. Runnable r = Foo::f; Runnable runnable = Foo::f; Including variable target name would render this change (previously recoverable and correct) a deserialiation failure If the user has changed the name, it seems reasonable to treat that as possibly meaning something else. Target type Predicate p = String::isEmpty; Function p = String::isEmpty; Including target type reduces the space of potential sequence number collisions. If you've changed the target type, it is a different lambda. This list is not exhaustive, and there are others we might consider. (For example, for lambdas that appear in method invocation context rather than assignment context, we might include the hash of the invoked method name and signature, or even the parameter index or name. This is where it starts to exhibit diminishing returns and increasing brittleness.) Taken in total, the effect is: * All order-of-capture issues become type-1 failures, rather than type-2 failures (modulo hash collisions). * Order of declaration issues are still present, but they are dramatically reduced, turning many type-2 failures into type-1 failures. * Some new type-1 failures are introduced, mostly those deriving from rename-refactors. The remaining type-2 failures could be dealt with if we added named lambdas in the future. (They are also prevented if users always assign lambdas to local variables whose names are unique within the method; in this way, the local-variable trick becomes a sort of poor-man's named lambda.) We can reduce the probability of collision further by using a different (and simpler) scheme for non-serializable lambdas (lambda$nnn), so that serializable lambdas can only accidentally collide with each other. However, there are some transformations which we will still not be able to avoid under this scheme. For example: *Old code** * *New code** * *Result** * Supplier s = foo ? () -> 1 : () -> 2; Supplier s = !foo ? () -> 2 : () -> 1; This change is behaviorally compatible but could result in type-2 failure, since both lambdas have the same target type, capture arity, etc. However^2, we can still detect this risk and warn the user. If for any /mmm/, we issue more than one sequence number /nnn/, we are at risk for a type-2 failure, and can issue a lint warning in that case, suggesting the user refactor to something more stable. (Who knows what that diagnostic message will look like.) With all the hash information above, it seems likely that the number of potentially colliding lambdas will be small enough that this warning would not come along too often. The impact of this change in the implementation is surprisingly small. It does not affect the serialized form (java.lang.invoke.SerializedLambda), or the generated deserialization code ($deserialize$). It only affects the code which generates the lambda method name, which needs access to a small additional bit of information -- the assignment target name. Similarly, detecting the condition required for warning is easy -- "sequence number != 1". Qualitatively, the result is still similar in feel to inner classes -- you can make "irrelevant" changes but we make no heroic attempts to recover from things like changes in capture order -- but we do a better job of detecting them (and, if you follow some coding discipline, you can avoid them entirely.) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130819/48ba02a4/attachment-0001.html From brian.goetz at oracle.com Mon Aug 19 08:58:30 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 19 Aug 2013 11:58:30 -0400 Subject: One final stab at improving lambda serialization In-Reply-To: <52123BA4.4020700@oracle.com> References: <52123BA4.4020700@oracle.com> Message-ID: <521240A6.6010604@oracle.com> Those of you receiving this in plain text (which probably is everyone on the -observers lists) will have some trouble deciphering. Mailman helpfully put the attachment here but not in a very useful format: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130819/48ba02a4/attachment-0001.html I've uploaded a more readable copy to: http://cr.openjdk.java.net/~briangoetz/eg-attachments/lambda-serialization.html On 8/19/2013 11:37 AM, Brian Goetz wrote: > *Background* > > The fundamental challenge with serialization is that the code that > defined a class at serialization time may have changed by the time > deserialization happens. Serialization is defined to be tolerant of > change to a certain extent, and admits a degree of customization to > allow additional flexibility. > > ... From dl at cs.oswego.edu Mon Aug 19 10:43:50 2013 From: dl at cs.oswego.edu (Doug Lea) Date: Mon, 19 Aug 2013 13:43:50 -0400 Subject: One final stab at improving lambda serialization In-Reply-To: <52123BA4.4020700@oracle.com> References: <52123BA4.4020700@oracle.com> Message-ID: <52125956.9040200@cs.oswego.edu> This is the sort of scheme I had in mind in my reply to David Lloyd (that might have in part led to this effort). And the details seem reasonable to me. Looks good to go, assuming no unexpected implementation snags. -Doug On 08/19/2013 11:37 AM, Brian Goetz wrote: > *Background* > > The fundamental challenge with serialization is that the code that defined a > class at serialization time may have changed by the time deserialization > happens. Serialization is defined to be tolerant of change to a certain extent, > and admits a degree of customization to allow additional flexibility. > > For ordinary classes, there are three lines of defense: > > * serialVersionUID > * serialization hooks > * default schema evolution > > Serial version UID for the target class must match exactly. By default, > serialization uses a serial version UID which is a hash of the classes > signatures. So this default approach means "any significant change to the > structure of the class (adding new methods, changing method or field signatures, > etc) renders serialized forms invalid". It is a common practice to explicitly > assign a serial version UID to a class, thereby disabling this mechanism. > > Classes that expect to evolve over time may use readObject/writeObject and/or > readResolve/writeReplace to customize the mapping between object state and > bytestream. If classes do not use this mechanism, serialization uses a default > schema evolution mechanism to adjust for changes in fields between serialization > and deserialization time; fields that are present in the bytestream but not in > the target class are ignored, and fields that are present in the target class > but not the bytestream get default values (zero, null, etc.) > > Anonymous classes follow the same approach and have access to the same > mechanisms (serialVersionUID, read/writeObject, etc), but they have two > additional sources of instability: > > * The name is generated as EnclosingClass$nnn. Any change to the set of > anonymous classes in the enclosing class may cause sequence numbers to change. > * The number and type of fields (appears in bytecode but not source code) are > generated based on the set of captured values. Any change to the set or > order of captured values can cause these signatures to change (in an > unspecified way). > > If the signatures remain stable, anonymous classes can use serialization hooks > to customize the serialized form, just like named classes. > > The EG has observed that users have largely learned to deal with the problems of > serialization of inner classes, either by (a) don't do it, or (b) ensure that > essentially the same bits are present on both sides of the pipe, preventing skew > from causing instability in either class names or signatures. > > The EG has set, as a minimum bar, that lambda serialization be "at least as good > as" anonymous class serialization. (This is not a high bar.) Further, the EG > has concluded that gratuitous deviations from anonymous class serialization are > undesirable, because, if users have to deal with an imperfect scheme, having > them deal with something that is basically the same as an imperfect scheme > they've already gotten used to is preferable to dealing with a new and > different scheme. > > Further, the EG has rejected the idea of arbitrarily restricting access to > serialization just because it is dangerous; users who have learned to use it > safely should not be unduly encumbered. > > *Failure modes > * > > For anonymous classes, one of two things will happen when attempting to > deserialize after things have changed "too much": > > 1. A deserialization failure due to either the name or signature not matching, > resulting in NoSuchMethodError, IncompatibleClassChangeError, etc. > 2. Deserializing to the wrong thing, without any evidence of error. > > Obviously, a type-2 failure is far worse than a type-1 failure, because no error > is raised and an unintended computation is performed. Here are two examples of > changes that are behaviorally compatible but which will result in type-2 > failures. The first has to do with order-of-declaration. > > *Old code** > * *New code** > * *Result** > * > Runnable r1 = new Runnable() { > void run() { > System.out.println("one"); > } > }; > Runnable r2 = new Runnable() { > void run() { > System.out.println("two"); > } > }; > Runnable r2 = new Runnable() { > void run() { > System.out.println("two"); > } > }; > Runnable r1 = new Runnable() { > void run() { > System.out.println("one"); > } > }; > Deserialized r1 (across skew) prints "two". > > This fails because in both cases, we get classes called Foo$1 and Foo$2, but in > the old code, these correspond to r1 and r2, but in the new code, these > correspond to r2 and r1. > > The other failure has to do with order-of-capture. > > *Old code** > * *New code** > * *Result** > * > String s1 = "foo"; > String s2 = "bar"; > Runnable r = new Runnable() { > void run() { > foo(s1, s2); > } > }; > > String s1 = "foo"; > String s2 = "bar"; > Runnable r = new Runnable() { > void run() { > String s = s2; > foo(s1, s); > } > }; > On deserialization, s1 and s2 are effectively swapped. > > This fails because the order of arguments in the implicitly generated > constructor of the inner class changes due to the order in which the compiler > encounters captured variables. If the reordered variables were of different > types, this would cause a type-1 failure, but if they are the same type, it > causes a type-2 failure. > > *User expectations* > > While experienced users are quick to state the "same bits on both sides" rule > for reliable deserialization, a bit of investigation reveals that user > expectations are actually higher than that. For example, if the compiler > generated a /random/ name for each lambda at compile time, then recompiling the > same source with the same compiler, and using the result for deserialization, > would fail. This is too restrictive; user expectations are not tied to "same > bits", but to a vaguer notion of "I compiled essentially the same source with > essentially the same compiler, and therefore didn't change anything > significant." For example, users would balk if adding a comment or changing > whitespace were to affect deserialization. Users likely expect (in part, due to > behavior of anonymous classes) changes to code that doesn't affect the lambda > directly or indirectly (e.g., add or remove a debugging println) also would not > affect the serialized form. > > In the absence of the user being able to explicitly name the lambda /and/ its > captures (as C++ does), there is no perfect solution. Instead, our goal can > only be to minimize type-2 failures while not unduly creating type-1 failures > when "no significant code change" happened. This means we have to put a stake > in the ground as to what constitutes "significant" code change. > > The de-facto (and likely accidental) definition of "significant" used by inner > classes here is: > > * Adding, removing, or reordering inner class instances earlier in the source > file; > * Changes to the number, order, or type of captured arguments > > This permits changes to code that has nothing to do with inner classes, and many > common refactorings as long as they do not affect the order of inner class > instances or their captures. > > *Current Lambda behavior* > > Lambda serialization currently behaves very similarly to anonymous class > serialization. Where anonymous classes have stable method names but unstable > class names, lambdas are the dual; unstable method names but stable class > names. But since both are used together, the resulting naming stability is > largely the same. > > We do one thing to increase naming stability for lambdas: we hash the name and > signature of the enclosing method in the lambda name. This insulates lambda > naming from the addition, removal, or reordering of methods within a class file, > but naming stability remains sensitive to the order of lambdas within the > method. Similarly, order-of-capture issues are largely similar to inner classes. > > Lambdas bodies are desugared to methods named in the following form: > lambda$/mmm/$/nnn/, where /mmm/ is a hash of the method name and signature, and > /nnn/ is a sequence number of lambdas that have the same /mmm/ hash. > > Because lambdas are instantiated via invokedynamic rather than invoking a > constructor directly, there is also slightly more leniency to changes to the > /types/ of captured argument; changing a captured argument from, say, String to > Object, would be a breaking change for anonymous classes (it changes the > constructor signature) but not for lambdas. This leniency is largely an > accidental artifact of translation, rather than a deliberate design decision. > > *Possible improvements* > > We can start by recognizing the role of the hash of the enclosing method in the > lambda method name. This reduces the set of lambdas that could collide from > "all the lambdas in the file" to "all the lambdas in the method." This reduces > the set of changes that cause both type-1 and type-2 errors. > > An additional observation is that there is a tension between trying to /recover > from/ skew (rather than simply trying to detect it, and failing deserialization) > and complexity. So I think we should focus primarily on detecting skew and > failing deserialization (turning type-2 failures into type-1) while at the same > time not unduly increasing the set of changes that cause type-1 errors, with the > goal of settling on an informal guideline of what constitutes "too much" change. > > We can do this by increasing the number of things that affect the /mmm/ hash, > effectively constructing the lambda-equivalent of the serialization version > UID. The more context we add to this hash, the smaller the set of lambdas that > hash to the same bucket gets, which reduces the space of possible collisions. > The following table shows possible candidates for inclusion, along with examples > of code that illustrate dependence on this item. > > *Item** > * *Old Code** > ------------------------------ > * *New Code** > **------------------------------* > *Effect** > * *Rationale** > * > Names of captured arguments > int x = ... > f(() -> x); > int y = ... > f(() -> y); Including the names of captured arguments in the hash would cause > rename-refactors of captured arguments to be considered a serialization-breaking > change. > While alpha-renaming is generally considered to be semantic-preserving, > serialization has always keyed off of names (such as field names) as being clues > to developer intent. It seems reasonable to say "If you change the names > involved, we have to assume a semantic change occurred." We cannot tell if a > name change is a simple alpha-rename or capturing a completely different > variable, so this is erring on the safe side. > Types of captured arguments > String x = ... > f(() -> x); Object x = ... > f(() -> x); > It seems reasonable to say that, if you capture arguments of a different type, > you've made a semantic change. > Order of captured arguments > () -> { > int a = f(x); > int b = g(y); > return h(a,b); > }; > () -> { > int b = g(y); > int a = f(x); > return h(a,b); > }; Changing the order of capture would become a type-1 failure rather than > possibly a type-2 failure. > Since we cannot detect whether the ordering change is semantically meaningful > or not, it is best to be conservative and say: change to capture order is likely > a semantic change. > Variable assignment target (if present) > Runnable r1 = Foo::f; > Runnable r2 = Foo::g; > Runnable r2 = Foo::g; > Runnable r1 = Foo::f; > > Including variable target name would render this reordering recoverable and correct > If the user has gone to the effort of providing a name, we can use this as a > hint to the meaning of the lambda. > > Runnable r = Foo::f; Runnable runnable = Foo::f; Including variable target > name would render this change (previously recoverable and correct) a > deserialiation failure > If the user has changed the name, it seems reasonable to treat that as possibly > meaning something else. > Target type > Predicate p = String::isEmpty; > Function p = String::isEmpty; Including target type reduces > the space of potential sequence number collisions. > If you've changed the target type, it is a different lambda. > > This list is not exhaustive, and there are others we might consider. (For > example, for lambdas that appear in method invocation context rather than > assignment context, we might include the hash of the invoked method name and > signature, or even the parameter index or name. This is where it starts to > exhibit diminishing returns and increasing brittleness.) > > Taken in total, the effect is: > > * All order-of-capture issues become type-1 failures, rather than type-2 > failures (modulo hash collisions). > * Order of declaration issues are still present, but they are dramatically > reduced, turning many type-2 failures into type-1 failures. > * Some new type-1 failures are introduced, mostly those deriving from > rename-refactors. > > The remaining type-2 failures could be dealt with if we added named lambdas in > the future. (They are also prevented if users always assign lambdas to local > variables whose names are unique within the method; in this way, the > local-variable trick becomes a sort of poor-man's named lambda.) > > We can reduce the probability of collision further by using a different (and > simpler) scheme for non-serializable lambdas (lambda$nnn), so that serializable > lambdas can only accidentally collide with each other. > > However, there are some transformations which we will still not be able to avoid > under this scheme. For example: > > *Old code** > * *New code** > * *Result** > * > Supplier s = > foo ? () -> 1 > : () -> 2; > Supplier s = > !foo ? () -> 2 > : () -> 1; This change is behaviorally compatible but could result in > type-2 failure, since both lambdas have the same target type, capture arity, etc. > > However^2, we can still detect this risk and warn the user. If for any /mmm/, > we issue more than one sequence number /nnn/, we are at risk for a type-2 > failure, and can issue a lint warning in that case, suggesting the user refactor > to something more stable. (Who knows what that diagnostic message will look > like.) With all the hash information above, it seems likely that the number of > potentially colliding lambdas will be small enough that this warning would not > come along too often. > > The impact of this change in the implementation is surprisingly small. It does > not affect the serialized form (java.lang.invoke.SerializedLambda), or the > generated deserialization code ($deserialize$). It only affects the code which > generates the lambda method name, which needs access to a small additional bit > of information -- the assignment target name. Similarly, detecting the > condition required for warning is easy -- "sequence number != 1". > > Qualitatively, the result is still similar in feel to inner classes -- you can > make "irrelevant" changes but we make no heroic attempts to recover from things > like changes in capture order -- but we do a better job of detecting them (and, > if you follow some coding discipline, you can avoid them entirely.) > > From dl at cs.oswego.edu Mon Aug 19 17:07:58 2013 From: dl at cs.oswego.edu (Doug Lea) Date: Mon, 19 Aug 2013 20:07:58 -0400 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <520D3952.30208@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> <520D3952.30208@oracle.com> Message-ID: <5212B35E.9070703@cs.oswego.edu> On 08/15/2013 04:25 PM, Brian Goetz wrote: > Last chance for comment. Sorry for delay. (Swamped...) My sense is that your AutpCloseable update could still do better in separating the underlying issue (that an object may hold resources) from the usage guidance ("must close" or not.) Here's a shot at it: /** * An object that may hold resources (such as file or socket handles) * until it is closed. The {@link #close()} method is called * automatically when exiting a {@code try}-with-resources block for * which the resource has been declared in the header. This * construction ensures prompt release of resources, avoiding resource * exhaustion exceptions and errors that may otherwise occur. * * @apiNote *

It is possible, and in fact common, for a base class to * implement AutoCloseable even though not all of its subclasses * (perhaps only a few) hold releasable resources. It is best practice * for usages applying to any possible subclass to use {@code * try}-with-resources constructions, but those applying only to sets * of classes that do not hold releasable resources need not do so. * In frameworks such as {@link java.util.Stream}, that support both * IO-based and non-IO-based forms, it is unnecessary to use {@code * try}-with-resources blocks in methods that will never be applied to * IO-based forms. * * @author Josh Bloch * @since 1.7 */ @FunctionalInterface public interface AutoCloseable { > > On 8/13/2013 1:03 PM, Brian Goetz wrote: >> OK, here's proposed spec: >> >> For AC: >> >> /** >> - * A resource that must be closed when it is no longer needed. >> + * A resource that may need to be closed when it is no longer needed, >> and whose >> + * {@link #close()} method will be called automatically when exiting a >> + * {@code try}-with-resources block for which the resource has been >> declared in >> + * the header. A resource would need to be explicitly closed if it >> holds a >> + * scarce system resource that is not promptly reclaimed by garbage >> collection, >> + * such as a file or socket handle. >> + * >> + * @apiNote >> + *

It should generally be possible to determine statically whether a >> resource >> + * that may need to be closed actually does need to be closed (in which >> case one >> + * should consider using try-with-resources to ensure its timely closure). >> + * For example, most {@link InputStream} implementations need to be >> closed, >> + * though {@link ByteArrayInputStream} does not need to be; most {@link >> Stream} >> + * implementations do not need to be closed, but streams which >> encapsulate >> + * IO resources, like those produced by the static factories in {@link >> Files} >> + * such as {@link Files#walk(Path, FileVisitOption...)}, do need to be >> closed. >> >> For BaseStream and friends: >> >> + *

Streams have a {@link #close()} method and implement {@link >> AutoCloseable}, >> + * but nearly all stream instances do not actually need to be closed >> after use. >> + * Generally, only streams whose source is an IO channel (such as those >> returned >> + * by {@link Files#lines(Path, Charset)}) will require closing. Most >> streams >> + * are backed by collections, arrays, or generating functions, which >> require no >> + * special resource management. (If a stream does require closing, it >> can be >> + * declared as a resource in a {@code try}-with-resources statement.) >> + * >> >> For Files.walk and friends: >> >> + *

The returned stream encapsulates one or more {@link >> DirectoryStream}s. >> + * If timely disposal of file system resources is required, the >> + * {@code try}-with-resources construct should be used to ensure >> that the >> + * stream's {@link Stream#close close} method is invoked after the >> stream >> + * operations are completed. >> >> >> On 8/12/2013 1:43 PM, Brian Goetz wrote: >>> After discussion both in the EG and within Oracle, here's where I think >>> this should land: >>> >>> - Eliminate MayHoldCloseableResource, which is fundamentally flawed >>> unless the spec for AutoCloseable is fixed, in which case it becomes >>> unnecessary. >>> - Fix the spec for AutoCloseable. >>> - Eliminate @HoldsResource, not because it is intrinsically bad, but >>> because it is a half solution and we should work towards a complete one >>> for 9, likely using something like JSR-308 checkers. The Eclipse >>> inspector can be adjusted to work with the new APIs. >>> - Make BaseStream implement AutoCloseable, with additional verbiage >>> regarding closing. >>> - Add additional spec to Files.{walk,lines,etc}, and to combinators >>> like concat, regarding their close behavior. >>> >>> The thing that pushed me over the edge from the various "work around the >>> problem" solutions is the existence of flatMap: >>> >>> Files.walk(dir) >>> .flatMap(Files::lines) >>> ... >>> >>> where each entry resulting from walk() will turn into a stream creation >>> and traversal within the flatMap implementation, and flatMap itself >>> needs a way to ensure it is deterministically closed. This pushes >>> strongly for having a close() method on Stream, which, if you're going >>> to have that, you might as well be AutoCloseable to make life easier for >>> users. >>> >>> The sole challenge here is to keep people from thinking that they *need* >>> to close all streams, and therefore mangle their code in bad ways to >>> satisfy this perceived need. >>> >>> >>> On 8/2/2013 7:39 PM, Brian Goetz wrote: >>>> There seems to be consensus that the current strawman for >>>> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >>>> >>>> Flaw #1 relates to @HoldsResource. While there seems to be consensus >>>> that annotations are reasonable for imparting information like this >>>> which is suited to static analysis tools, the current proposal (a) >>>> confusingly leaks through into subclass Javadoc and (b) seems too >>>> weak. Conclusion: just dropping @HR from the current scheme (leaving >>>> only MHCR <: AC) is an improvement; we can revisit what the correct >>>> annotations for hinting at resourceful-ness later. >>>> >>>> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >>>> overspecified; it says "a resource that must be closed." This is an >>>> overreach; what j.l.AC should say is "has a close method which is >>>> called automatically on exit from a TWR block." Because of this >>>> overspecify, subclassing doesn't help, because if an AC must be >>>> closed, so must a MHCR. Conclusion: dropping MHCR and just having >>>> BaseStream <: AC is an improvement. >>>> >>>> Flaw #3 relates to the spec of AC. The conclusion of the core library >>>> maintainers is that it is acceptable to correct the spec, and replace >>>> the "must" with "may". This leaves us with a fixed AC, and BaseStream >>>> <: AC. This seems strictly better than where we are now. Let's call >>>> this "Option 1." >>>> >>>> But, it is possible we might still choose to punt further for now. >>>> >>>> Observation: the need for Stream to have a close() method comes about >>>> because of the desire to have static factory methods that will create >>>> streams that encapsulate a resource that requires closing, when there >>>> is no way to get access to that underlying resource. Such as >>>> Files.walk(dir) or Files.lines(path). >>>> >>>> For other cases, like BufferedReader.lines(), the BR itself is AC, so >>>> users can handle resource management like: >>>> >>>> try (BufferedReader b = ...) { >>>> b.lines()... >>>> } >>>> >>>> and there is no need to close the *stream* at all, since the user has >>>> access to the actual closeable resource. It is only the static >>>> methods such as Files.list(dir), which don't provide the user access >>>> in this way. The methods we provide are: >>>> >>>> Files.list(Path) >>>> Files.walk(Path, maxDepth, options) >>>> Files.walk(Path, options) >>>> Files.find(Path, maxDepth, BiPredicate, >>>> options) >>>> Files.lines(Path) >>>> >>>> Colin @ Google pointed out that if the spliterators for these streams >>>> were lazy (didn't open the resource until the terminal began), and all >>>> terminal ops called the close hook in a finally block after >>>> processing, then we wouldn't need to be AC, since there would be no >>>> way to leak a closeable resource. (This would move resource-not-found >>>> exceptions from the factory method to the terminal op.) >>>> >>>> Still, having a close() method and *not* being AC may look weird to >>>> the users. >>>> >>>> Let's call this "Option 2": have a close() method, don't extend AC, >>>> make sure that all terminals call close, make sure that all static >>>> resource-holding stream factories produce late-binding spliterators. >>>> >>>> Option 3 would be to simply get rid of these methods (note that >>>> Files.list and Files.lines are pretty simple wrappers around existing >>>> AC abstractions, DirectoryStream and BufferedReader.) >>>> >>>> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >>>> >>>> > From joe.bowbeer at gmail.com Mon Aug 19 17:43:39 2013 From: joe.bowbeer at gmail.com (Joe Bowbeer) Date: Mon, 19 Aug 2013 17:43:39 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <5212B35E.9070703@cs.oswego.edu> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> <520D3952.30208@oracle.com> <5212B35E.9070703@cs.oswego.edu> Message-ID: Agree. Though [need not do so] seems awkward below: "... but those applying only to sets of classes that do not hold releasable resources [need not do so]." Try "do not need to"? On Mon, Aug 19, 2013 at 5:07 PM, Doug Lea

wrote: > On 08/15/2013 04:25 PM, Brian Goetz wrote: > >> Last chance for comment. >> > > Sorry for delay. (Swamped...) > > My sense is that your AutpCloseable update could still do better in > separating the underlying issue (that an object may hold resources) > from the usage guidance ("must close" or not.) Here's a shot at it: > > > /** > * An object that may hold resources (such as file or socket handles) > * until it is closed. The {@link #close()} method is called > * automatically when exiting a {@code try}-with-resources block for > * which the resource has been declared in the header. This > * construction ensures prompt release of resources, avoiding resource > * exhaustion exceptions and errors that may otherwise occur. > * > * @apiNote > *

It is possible, and in fact common, for a base class to > * implement AutoCloseable even though not all of its subclasses > * (perhaps only a few) hold releasable resources. It is best practice > * for usages applying to any possible subclass to use {@code > * try}-with-resources constructions, but those applying only to sets > * of classes that do not hold releasable resources need not do so. > * In frameworks such as {@link java.util.Stream}, that support both > * IO-based and non-IO-based forms, it is unnecessary to use {@code > * try}-with-resources blocks in methods that will never be applied to > * IO-based forms. > * > * @author Josh Bloch > * @since 1.7 > */ > @FunctionalInterface > public interface AutoCloseable { > > > > >> On 8/13/2013 1:03 PM, Brian Goetz wrote: >> >>> OK, here's proposed spec: >>> >>> For AC: >>> >>> /** >>> - * A resource that must be closed when it is no longer needed. >>> + * A resource that may need to be closed when it is no longer needed, >>> and whose >>> + * {@link #close()} method will be called automatically when exiting a >>> + * {@code try}-with-resources block for which the resource has been >>> declared in >>> + * the header. A resource would need to be explicitly closed if it >>> holds a >>> + * scarce system resource that is not promptly reclaimed by garbage >>> collection, >>> + * such as a file or socket handle. >>> + * >>> + * @apiNote >>> + *

It should generally be possible to determine statically whether a >>> resource >>> + * that may need to be closed actually does need to be closed (in which >>> case one >>> + * should consider using try-with-resources to ensure its timely >>> closure). >>> + * For example, most {@link InputStream} implementations need to be >>> closed, >>> + * though {@link ByteArrayInputStream} does not need to be; most {@link >>> Stream} >>> + * implementations do not need to be closed, but streams which >>> encapsulate >>> + * IO resources, like those produced by the static factories in {@link >>> Files} >>> + * such as {@link Files#walk(Path, FileVisitOption...)}, do need to be >>> closed. >>> >>> For BaseStream and friends: >>> >>> + *

Streams have a {@link #close()} method and implement {@link >>> AutoCloseable}, >>> + * but nearly all stream instances do not actually need to be closed >>> after use. >>> + * Generally, only streams whose source is an IO channel (such as those >>> returned >>> + * by {@link Files#lines(Path, Charset)}) will require closing. Most >>> streams >>> + * are backed by collections, arrays, or generating functions, which >>> require no >>> + * special resource management. (If a stream does require closing, it >>> can be >>> + * declared as a resource in a {@code try}-with-resources statement.) >>> + * >>> >>> For Files.walk and friends: >>> >>> + *

The returned stream encapsulates one or more {@link >>> DirectoryStream}s. >>> + * If timely disposal of file system resources is required, the >>> + * {@code try}-with-resources construct should be used to ensure >>> that the >>> + * stream's {@link Stream#close close} method is invoked after the >>> stream >>> + * operations are completed. >>> >>> >>> On 8/12/2013 1:43 PM, Brian Goetz wrote: >>> >>>> After discussion both in the EG and within Oracle, here's where I think >>>> this should land: >>>> >>>> - Eliminate MayHoldCloseableResource, which is fundamentally flawed >>>> unless the spec for AutoCloseable is fixed, in which case it becomes >>>> unnecessary. >>>> - Fix the spec for AutoCloseable. >>>> - Eliminate @HoldsResource, not because it is intrinsically bad, but >>>> because it is a half solution and we should work towards a complete one >>>> for 9, likely using something like JSR-308 checkers. The Eclipse >>>> inspector can be adjusted to work with the new APIs. >>>> - Make BaseStream implement AutoCloseable, with additional verbiage >>>> regarding closing. >>>> - Add additional spec to Files.{walk,lines,etc}, and to combinators >>>> like concat, regarding their close behavior. >>>> >>>> The thing that pushed me over the edge from the various "work around the >>>> problem" solutions is the existence of flatMap: >>>> >>>> Files.walk(dir) >>>> .flatMap(Files::lines) >>>> ... >>>> >>>> where each entry resulting from walk() will turn into a stream creation >>>> and traversal within the flatMap implementation, and flatMap itself >>>> needs a way to ensure it is deterministically closed. This pushes >>>> strongly for having a close() method on Stream, which, if you're going >>>> to have that, you might as well be AutoCloseable to make life easier for >>>> users. >>>> >>>> The sole challenge here is to keep people from thinking that they *need* >>>> to close all streams, and therefore mangle their code in bad ways to >>>> satisfy this perceived need. >>>> >>>> >>>> On 8/2/2013 7:39 PM, Brian Goetz wrote: >>>> >>>>> There seems to be consensus that the current strawman for >>>>> MayHoldCloseableResoruce and @HoldsResource is too flawed to proceed. >>>>> >>>>> Flaw #1 relates to @HoldsResource. While there seems to be consensus >>>>> that annotations are reasonable for imparting information like this >>>>> which is suited to static analysis tools, the current proposal (a) >>>>> confusingly leaks through into subclass Javadoc and (b) seems too >>>>> weak. Conclusion: just dropping @HR from the current scheme (leaving >>>>> only MHCR <: AC) is an improvement; we can revisit what the correct >>>>> annotations for hinting at resourceful-ness later. >>>>> >>>>> Flaw #2 relates to MHCR. The real problem is that AutoCloseable is >>>>> overspecified; it says "a resource that must be closed." This is an >>>>> overreach; what j.l.AC should say is "has a close method which is >>>>> called automatically on exit from a TWR block." Because of this >>>>> overspecify, subclassing doesn't help, because if an AC must be >>>>> closed, so must a MHCR. Conclusion: dropping MHCR and just having >>>>> BaseStream <: AC is an improvement. >>>>> >>>>> Flaw #3 relates to the spec of AC. The conclusion of the core library >>>>> maintainers is that it is acceptable to correct the spec, and replace >>>>> the "must" with "may". This leaves us with a fixed AC, and BaseStream >>>>> <: AC. This seems strictly better than where we are now. Let's call >>>>> this "Option 1." >>>>> >>>>> But, it is possible we might still choose to punt further for now. >>>>> >>>>> Observation: the need for Stream to have a close() method comes about >>>>> because of the desire to have static factory methods that will create >>>>> streams that encapsulate a resource that requires closing, when there >>>>> is no way to get access to that underlying resource. Such as >>>>> Files.walk(dir) or Files.lines(path). >>>>> >>>>> For other cases, like BufferedReader.lines(), the BR itself is AC, so >>>>> users can handle resource management like: >>>>> >>>>> try (BufferedReader b = ...) { >>>>> b.lines()... >>>>> } >>>>> >>>>> and there is no need to close the *stream* at all, since the user has >>>>> access to the actual closeable resource. It is only the static >>>>> methods such as Files.list(dir), which don't provide the user access >>>>> in this way. The methods we provide are: >>>>> >>>>> Files.list(Path) >>>>> Files.walk(Path, maxDepth, options) >>>>> Files.walk(Path, options) >>>>> Files.find(Path, maxDepth, BiPredicate, >>>>> options) >>>>> Files.lines(Path) >>>>> >>>>> Colin @ Google pointed out that if the spliterators for these streams >>>>> were lazy (didn't open the resource until the terminal began), and all >>>>> terminal ops called the close hook in a finally block after >>>>> processing, then we wouldn't need to be AC, since there would be no >>>>> way to leak a closeable resource. (This would move resource-not-found >>>>> exceptions from the factory method to the terminal op.) >>>>> >>>>> Still, having a close() method and *not* being AC may look weird to >>>>> the users. >>>>> >>>>> Let's call this "Option 2": have a close() method, don't extend AC, >>>>> make sure that all terminals call close, make sure that all static >>>>> resource-holding stream factories produce late-binding spliterators. >>>>> >>>>> Option 3 would be to simply get rid of these methods (note that >>>>> Files.list and Files.lines are pretty simple wrappers around existing >>>>> AC abstractions, DirectoryStream and BufferedReader.) >>>>> >>>>> Orthogonal to Options 2 and 3 would be fixing the spec of AC. >>>>> >>>>> >>>>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130819/7f9128bf/attachment-0001.html From dl at cs.oswego.edu Tue Aug 20 03:40:27 2013 From: dl at cs.oswego.edu (Doug Lea) Date: Tue, 20 Aug 2013 06:40:27 -0400 Subject: Revisiting MayHoldCloseableResource In-Reply-To: References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> <520D3952.30208@oracle.com> <5212B35E.9070703@cs.oswego.edu> Message-ID: <5213479B.7060805@cs.oswego.edu> On 08/19/2013 08:43 PM, Joe Bowbeer wrote: > Agree. > > Though [need not do so] seems awkward below: > Thanks. Take two: /** * An object that may hold resources (such as file or socket handles) * until it is closed. The {@link #close()} method of an AutoCloseable * object is called automatically when exiting a {@code * try}-with-resources block for which the object has been declared in * the header. This construction ensures prompt release, avoiding * resource exhaustion exceptions and errors that may otherwise occur. * * @apiNote *

It is possible, and in fact common, for a base class to * implement AutoCloseable even though not all of its subclasses or * instances hold releasable resources. It is best practice for usages * applying to any possible subclass to use {@code * try}-with-resources constructions. However, in frameworks such as * {@link java.util.Stream} that support both IO-based and * non-IO-based forms, {@code try}-with-resources blocks are in * general unnecessary when using non-IO-based forms. * * @author Josh Bloch * @since 1.7 */ @FunctionalInterface public interface AutoCloseable { From brian.goetz at oracle.com Tue Aug 20 07:45:40 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 20 Aug 2013 10:45:40 -0400 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <5213479B.7060805@cs.oswego.edu> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> <520D3952.30208@oracle.com> <5212B35E.9070703@cs.oswego.edu> <5213479B.7060805@cs.oswego.edu> Message-ID: <52138114.9000108@oracle.com> Improvement! Iterating... > /** > * An object that may hold resources (such as file or socket handles) > * until it is closed. The {@link #close()} method of an AutoCloseable > * object is called automatically when exiting a {@code > * try}-with-resources block for which the object has been declared in > * the header. This construction ensures prompt release, avoiding > * resource exhaustion exceptions and errors that may otherwise occur. > * > * @apiNote > *

It is possible, and in fact common, for a base class to > * implement AutoCloseable even though not all of its subclasses or > * instances hold releasable resources. It is best practice for usages > * applying to any possible subclass to use {@code > * try}-with-resources constructions. I worry this can be too-easily misread as "it is best practice for any use of AutoCloseable to use try-with-resources". How about: For code that must operate in complete generality, or when it is statically known that the AutoCloseable instance does hold resources requiring release, it is recommended to use TWR constructions to ensure prompt resource release. However ... However, in frameworks such as > * {@link java.util.Stream} that support both IO-based and > * non-IO-based forms, {@code try}-with-resources blocks are in > * general unnecessary when using non-IO-based forms. > * > * @author Josh Bloch > * @since 1.7 > */ > @FunctionalInterface > public interface AutoCloseable { > From dl at cs.oswego.edu Tue Aug 20 16:35:16 2013 From: dl at cs.oswego.edu (Doug Lea) Date: Tue, 20 Aug 2013 19:35:16 -0400 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <52138114.9000108@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> <520D3952.30208@oracle.com> <5212B35E.9070703@cs.oswego.edu> <5213479B.7060805@cs.oswego.edu> <52138114.9000108@oracle.com> Message-ID: <5213FD34.6000609@cs.oswego.edu> On 08/20/2013 10:45 AM, Brian Goetz wrote: > Improvement! Iterating... > For code that must operate in complete generality, or when it is statically > known that the AutoCloseable instance does hold resources requiring release, it > is recommended to use TWR constructions to ensure prompt resource release. > However ... This might be more straighforward changing "statically known" to just "known". Otherwise some people will be thinking that there is some static type that would tell them which kind they have, which there isn't. Leading to... /** * An object that may hold resources (such as file or socket handles) * until it is closed. The {@link #close()} method of an AutoCloseable * object is called automatically when exiting a {@code * try}-with-resources block for which the object has been declared in * the header. This construction ensures prompt release, avoiding * resource exhaustion exceptions and errors that may otherwise occur. * * @apiNote *

It is possible, and in fact common, for a base class to * implement AutoCloseable even though not all of its subclasses or * instances hold releasable resources. For usages applying to * any possible subclass, or those applying to classes * requiring resource release, it is best practice to use {@code * try}-with-resources constructions. However, in frameworks such as * {@link java.util.Stream} that support both IO-based and * non-IO-based forms, {@code try}-with-resources blocks are in * general unnecessary when using non-IO-based forms. * From joe.bowbeer at gmail.com Tue Aug 20 17:06:00 2013 From: joe.bowbeer at gmail.com (Joe Bowbeer) Date: Tue, 20 Aug 2013 17:06:00 -0700 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <5213FD34.6000609@cs.oswego.edu> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> <520D3952.30208@oracle.com> <5212B35E.9070703@cs.oswego.edu> <5213479B.7060805@cs.oswego.edu> <52138114.9000108@oracle.com> <5213FD34.6000609@cs.oswego.edu> Message-ID: Looks good to me. On Tue, Aug 20, 2013 at 4:35 PM, Doug Lea

wrote: > On 08/20/2013 10:45 AM, Brian Goetz wrote: > >> Improvement! Iterating... >> > > For code that must operate in complete generality, or when it is >> statically >> known that the AutoCloseable instance does hold resources requiring >> release, it >> is recommended to use TWR constructions to ensure prompt resource release. >> However ... >> > > This might be more straighforward changing "statically known" to > just "known". > Otherwise some people will be thinking that there is some static type > that would tell them which kind they have, which there isn't. > > Leading to... > > > > /** > * An object that may hold resources (such as file or socket handles) > * until it is closed. The {@link #close()} method of an AutoCloseable > * object is called automatically when exiting a {@code > * try}-with-resources block for which the object has been declared in > * the header. This construction ensures prompt release, avoiding > * resource exhaustion exceptions and errors that may otherwise occur. > * > * @apiNote > *

It is possible, and in fact common, for a base class to > * implement AutoCloseable even though not all of its subclasses or > * instances hold releasable resources. For usages applying to > * any possible subclass, or those applying to classes > * requiring resource release, it is best practice to use {@code > * try}-with-resources constructions. However, in frameworks such as > > * {@link java.util.Stream} that support both IO-based and > * non-IO-based forms, {@code try}-with-resources blocks are in > * general unnecessary when using non-IO-based forms. > * > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130820/b5e7fe0d/attachment.html From andrey.breslav at jetbrains.com Wed Aug 21 04:17:11 2013 From: andrey.breslav at jetbrains.com (Andrey Breslav) Date: Wed, 21 Aug 2013 15:17:11 +0400 Subject: One final stab at improving lambda serialization In-Reply-To: <52123BA4.4020700@oracle.com> References: <52123BA4.4020700@oracle.com> Message-ID: <1B173070-3CCE-4CA8-9A76-293F8FA1A1E7@jetbrains.com> > Qualitatively, the result is still similar in feel to inner classes -- you can make "irrelevant" changes but we make no heroic attempts to recover from things like changes in capture order -- but we do a better job of detecting them (and, if you follow some coding discipline, you can avoid them entirely.) > I think it is a good balance. I'll be glad to see it implemented this way. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130821/532f43d9/attachment.html From brian.goetz at oracle.com Wed Aug 21 11:20:32 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 21 Aug 2013 14:20:32 -0400 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <5213FD34.6000609@cs.oswego.edu> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> <520D3952.30208@oracle.com> <5212B35E.9070703@cs.oswego.edu> <5213479B.7060805@cs.oswego.edu> <52138114.9000108@oracle.com> <5213FD34.6000609@cs.oswego.edu> Message-ID: <521504F0.4060108@oracle.com> > This might be more straighforward changing "statically known" to > just "known". > Otherwise some people will be thinking that there is some static type > that would tell them which kind they have, which there isn't. OK, did that, integrating my sentence about "generality" with your latest, below. I think they say the same thing but is a little harder to misread as "always use TWR". > /** > * An object that may hold resources (such as file or socket handles) > * until it is closed. The {@link #close()} method of an AutoCloseable > * object is called automatically when exiting a {@code > * try}-with-resources block for which the object has been declared in > * the header. This construction ensures prompt release, avoiding > * resource exhaustion exceptions and errors that may otherwise occur. > * > * @apiNote > *

It is possible, and in fact common, for a base class to > * implement AutoCloseable even though not all of its subclasses or > * instances will hold releasable resources. For code that must operate > * in complete generality, or when it is known that the AutoCloseable > * instance requires resource release, it is recommended to use {@code > * try}-with-resources constructions. However, when using facilities such as > * {@link java.util.Stream} that support both IO-based and > * non-IO-based forms, {@code try}-with-resources blocks are in > * general unnecessary when using non-IO-based forms. > * > From brian.goetz at oracle.com Wed Aug 21 11:28:16 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 21 Aug 2013 14:28:16 -0400 Subject: Comparator combinators In-Reply-To: <520BDF59.3080100@oracle.com> References: <520BDF59.3080100@oracle.com> Message-ID: <521506C0.1080007@oracle.com> Waiting for input on this. I'm torn. On the one hand, we'd be publishing a new, lambda-centric API that doesn't conform to our "safe overloading" rules, and for which users would have to use explicit lambdas in the cases where method references aren't practical. On the other, if we rename the primitive versions to comparingXxx/thenComparingXxx, we're creating a different sort of risk -- users will say comparing(Person::getAge) and get boxing when they didn't intend (this is comparing(Function), when they probably wanted comparingInt(ToIntFunction)). Though this would be easy for IDEs to detect and suggest a replacement. I think the responsible thing to do is still probably to disambiguate by method name -- comparingInt, thenComparingLong, etc. Which isn't pretty but seems to be the new equilibrium (overloading and inference are in opposition; adding more inference means we can tolerate less overloading.) If there's no input, an executive decision will be made... On 8/14/2013 3:49 PM, Brian Goetz wrote: > This may well be our last API loose end... > > We currently have a pile of Comparator combinators, all currently called > comparing() or thenComparing(). Regardless of whether we choose to go > forward with the simplified overloading plan, these overloads have a > problem: for implicit lambdas, we can't distinguish between > > comparing(T->U) > and > comparing(T->int) > > because we don't type-check the body of the lambda until we do overload > selection, and don't do overload selection based on whether there are > type errors in the body (this was roundly rejected as being too > brittle). So for lambdas like: > > comparing(x -> x.size()) > > we've got a circularity -- even under the older, more complex scheme. > > We'd thought perhaps that, if we adopted the heuristic that we can > eagerly give a type to method reference that refers to a non-overloaded > method (e.g., Person::getAge), then cases like > > comparing(Person::getAge) > > can be handled, and this might take some of the pressure off the problem. > > For lambdas (with either approach) you can always get what you want with > an explicit lambda: > > comparing((Person p) -> p.getAge()) > > since this can be type-checked early. > > So the question is, is this good enough, even though it falls afoul of > the overloading guidelines for implicit lambdas? Or, should we mangle > the names of the methods? > > This question comes down to whether we think its better to force > everyone to explicitly pick a method, but then supporting implicit lambdas: > > comparingInt(x -> x.size()) > > or forcing users to use non-ambigous method refs or explicit lambdas: > > comparing(Person::getAge) > or > comparing((Person p) -> p.getAge()) > > Which disambiguation approach is worse? > From tim at peierls.net Wed Aug 21 11:33:38 2013 From: tim at peierls.net (Tim Peierls) Date: Wed, 21 Aug 2013 14:33:38 -0400 Subject: Comparator combinators In-Reply-To: <521506C0.1080007@oracle.com> References: <520BDF59.3080100@oracle.com> <521506C0.1080007@oracle.com> Message-ID: Explicit comparingXxx, etc. is best. On Wed, Aug 21, 2013 at 2:28 PM, Brian Goetz wrote: > Waiting for input on this. > > I'm torn. On the one hand, we'd be publishing a new, lambda-centric API > that doesn't conform to our "safe overloading" rules, and for which users > would have to use explicit lambdas in the cases where method references > aren't practical. > > On the other, if we rename the primitive versions to > comparingXxx/thenComparingXxx, we're creating a different sort of risk -- > users will say > > comparing(Person::getAge) > > and get boxing when they didn't intend (this is comparing(Function **Integer>), when they probably wanted comparingInt(ToIntFunction<**Person>)). > Though this would be easy for IDEs to detect and suggest a replacement. > > I think the responsible thing to do is still probably to disambiguate by > method name -- comparingInt, thenComparingLong, etc. Which isn't pretty > but seems to be the new equilibrium (overloading and inference are in > opposition; adding more inference means we can tolerate less overloading.) > > If there's no input, an executive decision will be made... > > > > > On 8/14/2013 3:49 PM, Brian Goetz wrote: > >> This may well be our last API loose end... >> >> We currently have a pile of Comparator combinators, all currently called >> comparing() or thenComparing(). Regardless of whether we choose to go >> forward with the simplified overloading plan, these overloads have a >> problem: for implicit lambdas, we can't distinguish between >> >> comparing(T->U) >> and >> comparing(T->int) >> >> because we don't type-check the body of the lambda until we do overload >> selection, and don't do overload selection based on whether there are >> type errors in the body (this was roundly rejected as being too >> brittle). So for lambdas like: >> >> comparing(x -> x.size()) >> >> we've got a circularity -- even under the older, more complex scheme. >> >> We'd thought perhaps that, if we adopted the heuristic that we can >> eagerly give a type to method reference that refers to a non-overloaded >> method (e.g., Person::getAge), then cases like >> >> comparing(Person::getAge) >> >> can be handled, and this might take some of the pressure off the problem. >> >> For lambdas (with either approach) you can always get what you want with >> an explicit lambda: >> >> comparing((Person p) -> p.getAge()) >> >> since this can be type-checked early. >> >> So the question is, is this good enough, even though it falls afoul of >> the overloading guidelines for implicit lambdas? Or, should we mangle >> the names of the methods? >> >> This question comes down to whether we think its better to force >> everyone to explicitly pick a method, but then supporting implicit >> lambdas: >> >> comparingInt(x -> x.size()) >> >> or forcing users to use non-ambigous method refs or explicit lambdas: >> >> comparing(Person::getAge) >> or >> comparing((Person p) -> p.getAge()) >> >> Which disambiguation approach is worse? >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130821/be763c00/attachment-0001.html From spullara at gmail.com Wed Aug 21 12:00:20 2013 From: spullara at gmail.com (Sam Pullara) Date: Wed, 21 Aug 2013 12:00:20 -0700 Subject: Comparator combinators In-Reply-To: References: <520BDF59.3080100@oracle.com> <521506C0.1080007@oracle.com> Message-ID: I agree. I would much rather end up accidentally boxing than having to disambiguate everywhere. Sam On Aug 21, 2013, at 11:33 AM, Tim Peierls wrote: > Explicit comparingXxx, etc. is best. > > > On Wed, Aug 21, 2013 at 2:28 PM, Brian Goetz wrote: > Waiting for input on this. > > I'm torn. On the one hand, we'd be publishing a new, lambda-centric API that doesn't conform to our "safe overloading" rules, and for which users would have to use explicit lambdas in the cases where method references aren't practical. > > On the other, if we rename the primitive versions to comparingXxx/thenComparingXxx, we're creating a different sort of risk -- users will say > > comparing(Person::getAge) > > and get boxing when they didn't intend (this is comparing(Function), when they probably wanted comparingInt(ToIntFunction)). Though this would be easy for IDEs to detect and suggest a replacement. > > I think the responsible thing to do is still probably to disambiguate by method name -- comparingInt, thenComparingLong, etc. Which isn't pretty but seems to be the new equilibrium (overloading and inference are in opposition; adding more inference means we can tolerate less overloading.) > > If there's no input, an executive decision will be made... > > > > > On 8/14/2013 3:49 PM, Brian Goetz wrote: > This may well be our last API loose end... > > We currently have a pile of Comparator combinators, all currently called > comparing() or thenComparing(). Regardless of whether we choose to go > forward with the simplified overloading plan, these overloads have a > problem: for implicit lambdas, we can't distinguish between > > comparing(T->U) > and > comparing(T->int) > > because we don't type-check the body of the lambda until we do overload > selection, and don't do overload selection based on whether there are > type errors in the body (this was roundly rejected as being too > brittle). So for lambdas like: > > comparing(x -> x.size()) > > we've got a circularity -- even under the older, more complex scheme. > > We'd thought perhaps that, if we adopted the heuristic that we can > eagerly give a type to method reference that refers to a non-overloaded > method (e.g., Person::getAge), then cases like > > comparing(Person::getAge) > > can be handled, and this might take some of the pressure off the problem. > > For lambdas (with either approach) you can always get what you want with > an explicit lambda: > > comparing((Person p) -> p.getAge()) > > since this can be type-checked early. > > So the question is, is this good enough, even though it falls afoul of > the overloading guidelines for implicit lambdas? Or, should we mangle > the names of the methods? > > This question comes down to whether we think its better to force > everyone to explicitly pick a method, but then supporting implicit lambdas: > > comparingInt(x -> x.size()) > > or forcing users to use non-ambigous method refs or explicit lambdas: > > comparing(Person::getAge) > or > comparing((Person p) -> p.getAge()) > > Which disambiguation approach is worse? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130821/f4ca4e19/attachment.html From brian.goetz at oracle.com Wed Aug 21 12:13:47 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 21 Aug 2013 15:13:47 -0400 Subject: Comparator combinators In-Reply-To: References: <520BDF59.3080100@oracle.com> <521506C0.1080007@oracle.com> Message-ID: <5215116B.4040503@oracle.com> To clarify, you would rather disambiguate with the method name than have to provide additional type information (explicit lambda, type witnesses, or cast), right? On 8/21/2013 3:00 PM, Sam Pullara wrote: > I agree. I would much rather end up accidentally boxing than having to > disambiguate everywhere. > > Sam > > On Aug 21, 2013, at 11:33 AM, Tim Peierls > wrote: > >> Explicit comparingXxx, etc. is best. >> >> >> On Wed, Aug 21, 2013 at 2:28 PM, Brian Goetz > > wrote: >> >> Waiting for input on this. >> >> I'm torn. On the one hand, we'd be publishing a new, >> lambda-centric API that doesn't conform to our "safe overloading" >> rules, and for which users would have to use explicit lambdas in >> the cases where method references aren't practical. >> >> On the other, if we rename the primitive versions to >> comparingXxx/thenComparingXxx, we're creating a different sort of >> risk -- users will say >> >> comparing(Person::getAge) >> >> and get boxing when they didn't intend (this is >> comparing(Function), when they probably wanted >> comparingInt(ToIntFunction<__Person>)). Though this would be easy >> for IDEs to detect and suggest a replacement. >> >> I think the responsible thing to do is still probably to >> disambiguate by method name -- comparingInt, thenComparingLong, >> etc. Which isn't pretty but seems to be the new equilibrium >> (overloading and inference are in opposition; adding more >> inference means we can tolerate less overloading.) >> >> If there's no input, an executive decision will be made... >> >> >> >> >> On 8/14/2013 3:49 PM, Brian Goetz wrote: >> >> This may well be our last API loose end... >> >> We currently have a pile of Comparator combinators, all >> currently called >> comparing() or thenComparing(). Regardless of whether we >> choose to go >> forward with the simplified overloading plan, these overloads >> have a >> problem: for implicit lambdas, we can't distinguish between >> >> comparing(T->U) >> and >> comparing(T->int) >> >> because we don't type-check the body of the lambda until we do >> overload >> selection, and don't do overload selection based on whether >> there are >> type errors in the body (this was roundly rejected as being too >> brittle). So for lambdas like: >> >> comparing(x -> x.size()) >> >> we've got a circularity -- even under the older, more complex >> scheme. >> >> We'd thought perhaps that, if we adopted the heuristic that we can >> eagerly give a type to method reference that refers to a >> non-overloaded >> method (e.g., Person::getAge), then cases like >> >> comparing(Person::getAge) >> >> can be handled, and this might take some of the pressure off >> the problem. >> >> For lambdas (with either approach) you can always get what you >> want with >> an explicit lambda: >> >> comparing((Person p) -> p.getAge()) >> >> since this can be type-checked early. >> >> So the question is, is this good enough, even though it falls >> afoul of >> the overloading guidelines for implicit lambdas? Or, should >> we mangle >> the names of the methods? >> >> This question comes down to whether we think its better to force >> everyone to explicitly pick a method, but then supporting >> implicit lambdas: >> >> comparingInt(x -> x.size()) >> >> or forcing users to use non-ambigous method refs or explicit >> lambdas: >> >> comparing(Person::getAge) >> or >> comparing((Person p) -> p.getAge()) >> >> Which disambiguation approach is worse? >> >> > From forax at univ-mlv.fr Wed Aug 21 12:13:54 2013 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 21 Aug 2013 21:13:54 +0200 Subject: Comparator combinators In-Reply-To: <521506C0.1080007@oracle.com> References: <520BDF59.3080100@oracle.com> <521506C0.1080007@oracle.com> Message-ID: <52151172.3060704@univ-mlv.fr> On 08/21/2013 08:28 PM, Brian Goetz wrote: > Waiting for input on this. > > I'm torn. On the one hand, we'd be publishing a new, lambda-centric > API that doesn't conform to our "safe overloading" rules, and for > which users would have to use explicit lambdas in the cases where > method references aren't practical. > > On the other, if we rename the primitive versions to > comparingXxx/thenComparingXxx, we're creating a different sort of risk > -- users will say > > comparing(Person::getAge) > > and get boxing when they didn't intend (this is > comparing(Function), when they probably wanted > comparingInt(ToIntFunction)). Though this would be easy for > IDEs to detect and suggest a replacement. > > I think the responsible thing to do is still probably to disambiguate > by method name -- comparingInt, thenComparingLong, etc. Which isn't > pretty but seems to be the new equilibrium (overloading and inference > are in opposition; adding more inference means we can tolerate less > overloading.) > > If there's no input, an executive decision will be made... yes, disambiguate by method name seems the only reasonable choice given the way overloading now works. R?mi > > > > On 8/14/2013 3:49 PM, Brian Goetz wrote: >> This may well be our last API loose end... >> >> We currently have a pile of Comparator combinators, all currently called >> comparing() or thenComparing(). Regardless of whether we choose to go >> forward with the simplified overloading plan, these overloads have a >> problem: for implicit lambdas, we can't distinguish between >> >> comparing(T->U) >> and >> comparing(T->int) >> >> because we don't type-check the body of the lambda until we do overload >> selection, and don't do overload selection based on whether there are >> type errors in the body (this was roundly rejected as being too >> brittle). So for lambdas like: >> >> comparing(x -> x.size()) >> >> we've got a circularity -- even under the older, more complex scheme. >> >> We'd thought perhaps that, if we adopted the heuristic that we can >> eagerly give a type to method reference that refers to a non-overloaded >> method (e.g., Person::getAge), then cases like >> >> comparing(Person::getAge) >> >> can be handled, and this might take some of the pressure off the >> problem. >> >> For lambdas (with either approach) you can always get what you want with >> an explicit lambda: >> >> comparing((Person p) -> p.getAge()) >> >> since this can be type-checked early. >> >> So the question is, is this good enough, even though it falls afoul of >> the overloading guidelines for implicit lambdas? Or, should we mangle >> the names of the methods? >> >> This question comes down to whether we think its better to force >> everyone to explicitly pick a method, but then supporting implicit >> lambdas: >> >> comparingInt(x -> x.size()) >> >> or forcing users to use non-ambigous method refs or explicit lambdas: >> >> comparing(Person::getAge) >> or >> comparing((Person p) -> p.getAge()) >> >> Which disambiguation approach is worse? >> From spullara at gmail.com Wed Aug 21 13:24:27 2013 From: spullara at gmail.com (Sam Pullara) Date: Wed, 21 Aug 2013 13:24:27 -0700 Subject: Comparator combinators In-Reply-To: <5215116B.4040503@oracle.com> References: <520BDF59.3080100@oracle.com> <521506C0.1080007@oracle.com> <5215116B.4040503@oracle.com> Message-ID: <7DCF850A-4577-41F7-87DE-3A6EE5E633ED@gmail.com> Yes, I'd like to have it in the method name. Sam On Aug 21, 2013, at 12:13 PM, Brian Goetz wrote: > To clarify, you would rather disambiguate with the method name than have to provide additional type information (explicit lambda, type witnesses, or cast), right? > > On 8/21/2013 3:00 PM, Sam Pullara wrote: >> I agree. I would much rather end up accidentally boxing than having to >> disambiguate everywhere. >> >> Sam >> >> On Aug 21, 2013, at 11:33 AM, Tim Peierls > > wrote: >> >>> Explicit comparingXxx, etc. is best. >>> >>> >>> On Wed, Aug 21, 2013 at 2:28 PM, Brian Goetz >> > wrote: >>> >>> Waiting for input on this. >>> >>> I'm torn. On the one hand, we'd be publishing a new, >>> lambda-centric API that doesn't conform to our "safe overloading" >>> rules, and for which users would have to use explicit lambdas in >>> the cases where method references aren't practical. >>> >>> On the other, if we rename the primitive versions to >>> comparingXxx/thenComparingXxx, we're creating a different sort of >>> risk -- users will say >>> >>> comparing(Person::getAge) >>> >>> and get boxing when they didn't intend (this is >>> comparing(Function), when they probably wanted >>> comparingInt(ToIntFunction<__Person>)). Though this would be easy >>> for IDEs to detect and suggest a replacement. >>> >>> I think the responsible thing to do is still probably to >>> disambiguate by method name -- comparingInt, thenComparingLong, >>> etc. Which isn't pretty but seems to be the new equilibrium >>> (overloading and inference are in opposition; adding more >>> inference means we can tolerate less overloading.) >>> >>> If there's no input, an executive decision will be made... >>> >>> >>> >>> >>> On 8/14/2013 3:49 PM, Brian Goetz wrote: >>> >>> This may well be our last API loose end... >>> >>> We currently have a pile of Comparator combinators, all >>> currently called >>> comparing() or thenComparing(). Regardless of whether we >>> choose to go >>> forward with the simplified overloading plan, these overloads >>> have a >>> problem: for implicit lambdas, we can't distinguish between >>> >>> comparing(T->U) >>> and >>> comparing(T->int) >>> >>> because we don't type-check the body of the lambda until we do >>> overload >>> selection, and don't do overload selection based on whether >>> there are >>> type errors in the body (this was roundly rejected as being too >>> brittle). So for lambdas like: >>> >>> comparing(x -> x.size()) >>> >>> we've got a circularity -- even under the older, more complex >>> scheme. >>> >>> We'd thought perhaps that, if we adopted the heuristic that we can >>> eagerly give a type to method reference that refers to a >>> non-overloaded >>> method (e.g., Person::getAge), then cases like >>> >>> comparing(Person::getAge) >>> >>> can be handled, and this might take some of the pressure off >>> the problem. >>> >>> For lambdas (with either approach) you can always get what you >>> want with >>> an explicit lambda: >>> >>> comparing((Person p) -> p.getAge()) >>> >>> since this can be type-checked early. >>> >>> So the question is, is this good enough, even though it falls >>> afoul of >>> the overloading guidelines for implicit lambdas? Or, should >>> we mangle >>> the names of the methods? >>> >>> This question comes down to whether we think its better to force >>> everyone to explicitly pick a method, but then supporting >>> implicit lambdas: >>> >>> comparingInt(x -> x.size()) >>> >>> or forcing users to use non-ambigous method refs or explicit >>> lambdas: >>> >>> comparing(Person::getAge) >>> or >>> comparing((Person p) -> p.getAge()) >>> >>> Which disambiguation approach is worse? >>> >>> >> From henry.jen at oracle.com Wed Aug 21 13:53:42 2013 From: henry.jen at oracle.com (Henry Jen) Date: Wed, 21 Aug 2013 13:53:42 -0700 Subject: Comparator combinators In-Reply-To: <521506C0.1080007@oracle.com> References: <520BDF59.3080100@oracle.com> <521506C0.1080007@oracle.com> Message-ID: Seems the consensus is converging to disambiguate with the method name, which I am OK with. In fact, we had done that before and revert back because people don't like it. As mentioned several times, the not-intended boxing is the main concern. While I understand that IDE could help, I don't like to assume people will be using one(although most likely they are). How often will we need to add these explicit type? Is that type harder to figure out than knowing what we are comparing? Cheers, Henry On Aug 21, 2013, at 11:28 AM, Brian Goetz wrote: > Waiting for input on this. > > I'm torn. On the one hand, we'd be publishing a new, lambda-centric API that doesn't conform to our "safe overloading" rules, and for which users would have to use explicit lambdas in the cases where method references aren't practical. > > On the other, if we rename the primitive versions to comparingXxx/thenComparingXxx, we're creating a different sort of risk -- users will say > > comparing(Person::getAge) > > and get boxing when they didn't intend (this is comparing(Function), when they probably wanted comparingInt(ToIntFunction)). Though this would be easy for IDEs to detect and suggest a replacement. > > I think the responsible thing to do is still probably to disambiguate by method name -- comparingInt, thenComparingLong, etc. Which isn't pretty but seems to be the new equilibrium (overloading and inference are in opposition; adding more inference means we can tolerate less overloading.) > > If there's no input, an executive decision will be made... > > > > On 8/14/2013 3:49 PM, Brian Goetz wrote: >> This may well be our last API loose end... >> >> We currently have a pile of Comparator combinators, all currently called >> comparing() or thenComparing(). Regardless of whether we choose to go >> forward with the simplified overloading plan, these overloads have a >> problem: for implicit lambdas, we can't distinguish between >> >> comparing(T->U) >> and >> comparing(T->int) >> >> because we don't type-check the body of the lambda until we do overload >> selection, and don't do overload selection based on whether there are >> type errors in the body (this was roundly rejected as being too >> brittle). So for lambdas like: >> >> comparing(x -> x.size()) >> >> we've got a circularity -- even under the older, more complex scheme. >> >> We'd thought perhaps that, if we adopted the heuristic that we can >> eagerly give a type to method reference that refers to a non-overloaded >> method (e.g., Person::getAge), then cases like >> >> comparing(Person::getAge) >> >> can be handled, and this might take some of the pressure off the problem. >> >> For lambdas (with either approach) you can always get what you want with >> an explicit lambda: >> >> comparing((Person p) -> p.getAge()) >> >> since this can be type-checked early. >> >> So the question is, is this good enough, even though it falls afoul of >> the overloading guidelines for implicit lambdas? Or, should we mangle >> the names of the methods? >> >> This question comes down to whether we think its better to force >> everyone to explicitly pick a method, but then supporting implicit lambdas: >> >> comparingInt(x -> x.size()) >> >> or forcing users to use non-ambigous method refs or explicit lambdas: >> >> comparing(Person::getAge) >> or >> comparing((Person p) -> p.getAge()) >> >> Which disambiguation approach is worse? >> From spullara at gmail.com Wed Aug 21 14:18:05 2013 From: spullara at gmail.com (Sam Pullara) Date: Wed, 21 Aug 2013 14:18:05 -0700 Subject: Comparator combinators In-Reply-To: References: <520BDF59.3080100@oracle.com> <521506C0.1080007@oracle.com> Message-ID: <36839805328721330@unknownmsgid> I think the IntFunction version will be used more than any other version. We could probably name that one comparing and name the other ones with their types, e.g. comparingObject. Sam > On Aug 21, 2013, at 2:09 PM, Henry Jen wrote: > > Seems the consensus is converging to disambiguate with the method name, which I am OK with. In fact, we had done that before and revert back because people don't like it. > > As mentioned several times, the not-intended boxing is the main concern. While I understand that IDE could help, I don't like to assume people will be using one(although most likely they are). > > How often will we need to add these explicit type? Is that type harder to figure out than knowing what we are comparing? > > Cheers, > Henry > >> On Aug 21, 2013, at 11:28 AM, Brian Goetz wrote: >> >> Waiting for input on this. >> >> I'm torn. On the one hand, we'd be publishing a new, lambda-centric API that doesn't conform to our "safe overloading" rules, and for which users would have to use explicit lambdas in the cases where method references aren't practical. >> >> On the other, if we rename the primitive versions to comparingXxx/thenComparingXxx, we're creating a different sort of risk -- users will say >> >> comparing(Person::getAge) >> >> and get boxing when they didn't intend (this is comparing(Function), when they probably wanted comparingInt(ToIntFunction)). Though this would be easy for IDEs to detect and suggest a replacement. >> >> I think the responsible thing to do is still probably to disambiguate by method name -- comparingInt, thenComparingLong, etc. Which isn't pretty but seems to be the new equilibrium (overloading and inference are in opposition; adding more inference means we can tolerate less overloading.) >> >> If there's no input, an executive decision will be made... >> >> >> >>> On 8/14/2013 3:49 PM, Brian Goetz wrote: >>> This may well be our last API loose end... >>> >>> We currently have a pile of Comparator combinators, all currently called >>> comparing() or thenComparing(). Regardless of whether we choose to go >>> forward with the simplified overloading plan, these overloads have a >>> problem: for implicit lambdas, we can't distinguish between >>> >>> comparing(T->U) >>> and >>> comparing(T->int) >>> >>> because we don't type-check the body of the lambda until we do overload >>> selection, and don't do overload selection based on whether there are >>> type errors in the body (this was roundly rejected as being too >>> brittle). So for lambdas like: >>> >>> comparing(x -> x.size()) >>> >>> we've got a circularity -- even under the older, more complex scheme. >>> >>> We'd thought perhaps that, if we adopted the heuristic that we can >>> eagerly give a type to method reference that refers to a non-overloaded >>> method (e.g., Person::getAge), then cases like >>> >>> comparing(Person::getAge) >>> >>> can be handled, and this might take some of the pressure off the problem. >>> >>> For lambdas (with either approach) you can always get what you want with >>> an explicit lambda: >>> >>> comparing((Person p) -> p.getAge()) >>> >>> since this can be type-checked early. >>> >>> So the question is, is this good enough, even though it falls afoul of >>> the overloading guidelines for implicit lambdas? Or, should we mangle >>> the names of the methods? >>> >>> This question comes down to whether we think its better to force >>> everyone to explicitly pick a method, but then supporting implicit lambdas: >>> >>> comparingInt(x -> x.size()) >>> >>> or forcing users to use non-ambigous method refs or explicit lambdas: >>> >>> comparing(Person::getAge) >>> or >>> comparing((Person p) -> p.getAge()) >>> >>> Which disambiguation approach is worse? > From joe.bowbeer at gmail.com Wed Aug 21 14:23:21 2013 From: joe.bowbeer at gmail.com (Joe Bowbeer) Date: Wed, 21 Aug 2013 14:23:21 -0700 Subject: Comparator combinators In-Reply-To: <36839805328721330@unknownmsgid> References: <520BDF59.3080100@oracle.com> <521506C0.1080007@oracle.com> <36839805328721330@unknownmsgid> Message-ID: > We could probably name that one comparing and name the other ones with their types, e.g. comparingObject. This is the reverse of the way these are named elsewhere. To be consistent, you'd need to rename IntStream as Stream, and Stream as ObjectStream. I'd rather stick with an explicit Int where needed. On Wed, Aug 21, 2013 at 2:18 PM, Sam Pullara wrote: > I think the IntFunction version will be used more than any other > version. We could probably name that one comparing and name the other > ones with their types, e.g. comparingObject. > > Sam > > > On Aug 21, 2013, at 2:09 PM, Henry Jen wrote: > > > > Seems the consensus is converging to disambiguate with the method name, > which I am OK with. In fact, we had done that before and revert back > because people don't like it. > > > > As mentioned several times, the not-intended boxing is the main concern. > While I understand that IDE could help, I don't like to assume people will > be using one(although most likely they are). > > > > How often will we need to add these explicit type? Is that type harder > to figure out than knowing what we are comparing? > > > > Cheers, > > Henry > > > >> On Aug 21, 2013, at 11:28 AM, Brian Goetz > wrote: > >> > >> Waiting for input on this. > >> > >> I'm torn. On the one hand, we'd be publishing a new, lambda-centric > API that doesn't conform to our "safe overloading" rules, and for which > users would have to use explicit lambdas in the cases where method > references aren't practical. > >> > >> On the other, if we rename the primitive versions to > comparingXxx/thenComparingXxx, we're creating a different sort of risk -- > users will say > >> > >> comparing(Person::getAge) > >> > >> and get boxing when they didn't intend (this is > comparing(Function), when they probably wanted > comparingInt(ToIntFunction)). Though this would be easy for IDEs > to detect and suggest a replacement. > >> > >> I think the responsible thing to do is still probably to disambiguate > by method name -- comparingInt, thenComparingLong, etc. Which isn't pretty > but seems to be the new equilibrium (overloading and inference are in > opposition; adding more inference means we can tolerate less overloading.) > >> > >> If there's no input, an executive decision will be made... > >> > >> > >> > >>> On 8/14/2013 3:49 PM, Brian Goetz wrote: > >>> This may well be our last API loose end... > >>> > >>> We currently have a pile of Comparator combinators, all currently > called > >>> comparing() or thenComparing(). Regardless of whether we choose to go > >>> forward with the simplified overloading plan, these overloads have a > >>> problem: for implicit lambdas, we can't distinguish between > >>> > >>> comparing(T->U) > >>> and > >>> comparing(T->int) > >>> > >>> because we don't type-check the body of the lambda until we do overload > >>> selection, and don't do overload selection based on whether there are > >>> type errors in the body (this was roundly rejected as being too > >>> brittle). So for lambdas like: > >>> > >>> comparing(x -> x.size()) > >>> > >>> we've got a circularity -- even under the older, more complex scheme. > >>> > >>> We'd thought perhaps that, if we adopted the heuristic that we can > >>> eagerly give a type to method reference that refers to a non-overloaded > >>> method (e.g., Person::getAge), then cases like > >>> > >>> comparing(Person::getAge) > >>> > >>> can be handled, and this might take some of the pressure off the > problem. > >>> > >>> For lambdas (with either approach) you can always get what you want > with > >>> an explicit lambda: > >>> > >>> comparing((Person p) -> p.getAge()) > >>> > >>> since this can be type-checked early. > >>> > >>> So the question is, is this good enough, even though it falls afoul of > >>> the overloading guidelines for implicit lambdas? Or, should we mangle > >>> the names of the methods? > >>> > >>> This question comes down to whether we think its better to force > >>> everyone to explicitly pick a method, but then supporting implicit > lambdas: > >>> > >>> comparingInt(x -> x.size()) > >>> > >>> or forcing users to use non-ambigous method refs or explicit lambdas: > >>> > >>> comparing(Person::getAge) > >>> or > >>> comparing((Person p) -> p.getAge()) > >>> > >>> Which disambiguation approach is worse? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130821/5af19d02/attachment.html From spullara at gmail.com Wed Aug 21 14:24:39 2013 From: spullara at gmail.com (Sam Pullara) Date: Wed, 21 Aug 2013 14:24:39 -0700 Subject: Comparator combinators In-Reply-To: References: <520BDF59.3080100@oracle.com> <521506C0.1080007@oracle.com> <36839805328721330@unknownmsgid> Message-ID: <58AA32E1-7BBA-4756-B390-57F6E2A71CAF@gmail.com> It was more a commentary on how often I would have to complicate the lambda expression than an actual suggestion. But I agree we can't do it for consistency reasons. Sam On Aug 21, 2013, at 2:23 PM, Joe Bowbeer wrote: > > We could probably name that one comparing and name the other ones with their types, e.g. comparingObject. > > This is the reverse of the way these are named elsewhere. To be consistent, you'd need to rename IntStream as Stream, and Stream as ObjectStream. > > I'd rather stick with an explicit Int where needed. > > > On Wed, Aug 21, 2013 at 2:18 PM, Sam Pullara wrote: > I think the IntFunction version will be used more than any other > version. We could probably name that one comparing and name the other > ones with their types, e.g. comparingObject. > > Sam > > > On Aug 21, 2013, at 2:09 PM, Henry Jen wrote: > > > > Seems the consensus is converging to disambiguate with the method name, which I am OK with. In fact, we had done that before and revert back because people don't like it. > > > > As mentioned several times, the not-intended boxing is the main concern. While I understand that IDE could help, I don't like to assume people will be using one(although most likely they are). > > > > How often will we need to add these explicit type? Is that type harder to figure out than knowing what we are comparing? > > > > Cheers, > > Henry > > > >> On Aug 21, 2013, at 11:28 AM, Brian Goetz wrote: > >> > >> Waiting for input on this. > >> > >> I'm torn. On the one hand, we'd be publishing a new, lambda-centric API that doesn't conform to our "safe overloading" rules, and for which users would have to use explicit lambdas in the cases where method references aren't practical. > >> > >> On the other, if we rename the primitive versions to comparingXxx/thenComparingXxx, we're creating a different sort of risk -- users will say > >> > >> comparing(Person::getAge) > >> > >> and get boxing when they didn't intend (this is comparing(Function), when they probably wanted comparingInt(ToIntFunction)). Though this would be easy for IDEs to detect and suggest a replacement. > >> > >> I think the responsible thing to do is still probably to disambiguate by method name -- comparingInt, thenComparingLong, etc. Which isn't pretty but seems to be the new equilibrium (overloading and inference are in opposition; adding more inference means we can tolerate less overloading.) > >> > >> If there's no input, an executive decision will be made... > >> > >> > >> > >>> On 8/14/2013 3:49 PM, Brian Goetz wrote: > >>> This may well be our last API loose end... > >>> > >>> We currently have a pile of Comparator combinators, all currently called > >>> comparing() or thenComparing(). Regardless of whether we choose to go > >>> forward with the simplified overloading plan, these overloads have a > >>> problem: for implicit lambdas, we can't distinguish between > >>> > >>> comparing(T->U) > >>> and > >>> comparing(T->int) > >>> > >>> because we don't type-check the body of the lambda until we do overload > >>> selection, and don't do overload selection based on whether there are > >>> type errors in the body (this was roundly rejected as being too > >>> brittle). So for lambdas like: > >>> > >>> comparing(x -> x.size()) > >>> > >>> we've got a circularity -- even under the older, more complex scheme. > >>> > >>> We'd thought perhaps that, if we adopted the heuristic that we can > >>> eagerly give a type to method reference that refers to a non-overloaded > >>> method (e.g., Person::getAge), then cases like > >>> > >>> comparing(Person::getAge) > >>> > >>> can be handled, and this might take some of the pressure off the problem. > >>> > >>> For lambdas (with either approach) you can always get what you want with > >>> an explicit lambda: > >>> > >>> comparing((Person p) -> p.getAge()) > >>> > >>> since this can be type-checked early. > >>> > >>> So the question is, is this good enough, even though it falls afoul of > >>> the overloading guidelines for implicit lambdas? Or, should we mangle > >>> the names of the methods? > >>> > >>> This question comes down to whether we think its better to force > >>> everyone to explicitly pick a method, but then supporting implicit lambdas: > >>> > >>> comparingInt(x -> x.size()) > >>> > >>> or forcing users to use non-ambigous method refs or explicit lambdas: > >>> > >>> comparing(Person::getAge) > >>> or > >>> comparing((Person p) -> p.getAge()) > >>> > >>> Which disambiguation approach is worse? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/lambda-libs-spec-experts/attachments/20130821/62a0fc33/attachment-0001.html From dl at cs.oswego.edu Wed Aug 21 15:51:23 2013 From: dl at cs.oswego.edu (Doug Lea) Date: Wed, 21 Aug 2013 18:51:23 -0400 Subject: Revisiting MayHoldCloseableResource In-Reply-To: <521504F0.4060108@oracle.com> References: <824FB268-0649-4F69-972A-E93F77314D35@oracle.com> <52091ECF.2060204@oracle.com> <520A66F3.1020603@oracle.com> <520D3952.30208@oracle.com> <5212B35E.9070703@cs.oswego.edu> <5213479B.7060805@cs.oswego.edu> <52138114.9000108@oracle.com> <5213FD34.6000609@cs.oswego.edu> <521504F0.4060108@oracle.com> Message-ID: <5215446B.4050805@cs.oswego.edu> Fine with me. Let's claim success. -Doug On 08/21/2013 02:20 PM, Brian Goetz wrote: >> This might be more straighforward changing "statically known" to >> just "known". > > Otherwise some people will be thinking that there is some static type > > that would tell them which kind they have, which there isn't. > > OK, did that, integrating my sentence about "generality" with your latest, > below. I think they say the same thing but is a little harder to misread as > "always use TWR". > >> /** >> * An object that may hold resources (such as file or socket handles) >> * until it is closed. The {@link #close()} method of an AutoCloseable >> * object is called automatically when exiting a {@code >> * try}-with-resources block for which the object has been declared in >> * the header. This construction ensures prompt release, avoiding >> * resource exhaustion exceptions and errors that may otherwise occur. >> * >> * @apiNote >> *

It is possible, and in fact common, for a base class to >> * implement AutoCloseable even though not all of its subclasses or >> * instances will hold releasable resources. For code that must operate >> * in complete generality, or when it is known that the AutoCloseable > > * instance requires resource release, it is recommended to use {@code >> * try}-with-resources constructions. However, when using facilities such as >> * {@link java.util.Stream} that support both IO-based and >> * non-IO-based forms, {@code try}-with-resources blocks are in >> * general unnecessary when using non-IO-based forms. >> * >> > From Vladimir.Zakharov at gs.com Wed Aug 21 19:25:39 2013 From: Vladimir.Zakharov at gs.com (Zakharov, Vladimir) Date: Wed, 21 Aug 2013 22:25:39 -0400 Subject: One final stab at improving lambda serialization In-Reply-To: <52125956.9040200@cs.oswego.edu> References: <52123BA4.4020700@oracle.com> <52125956.9040200@cs.oswego.edu> Message-ID: I agree. Looks like this proposal strikes just the right balance: it is not being unnecessarily (and unexpectedly) restrictive while avoiding "silent" failures. I am supportive of serialization behavior implemented as described below. Thank you, Vlad The Goldman Sachs Group, Inc. All rights reserved. See http://www.gs.com/disclaimer/global_email for important risk disclosures, conflicts of interest and other terms and conditions relating to this e-mail and your reliance on information contained in it. This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.gs.com/disclaimer/email for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. -----Original Message----- From: lambda-libs-spec-experts-bounces at openjdk.java.net [mailto:lambda-libs-spec-experts-bounces at openjdk.java.net] On Behalf Of Doug Lea Sent: Monday, August 19, 2013 1:44 PM To: Brian Goetz Cc: lambda-libs-spec-experts at openjdk.java.net; lambda-spec-experts at openjdk.java.net Subject: Re: One final stab at improving lambda serialization This is the sort of scheme I had in mind in my reply to David Lloyd (that might have in part led to this effort). And the details seem reasonable to me. Looks good to go, assuming no unexpected implementation snags. -Doug On 08/19/2013 11:37 AM, Brian Goetz wrote: > *Background* > > The fundamental challenge with serialization is that the code that defined a > class at serialization time may have changed by the time deserialization > happens. Serialization is defined to be tolerant of change to a certain extent, > and admits a degree of customization to allow additional flexibility. > > For ordinary classes, there are three lines of defense: > > * serialVersionUID > * serialization hooks > * default schema evolution > > Serial version UID for the target class must match exactly. By default, > serialization uses a serial version UID which is a hash of the classes > signatures. So this default approach means "any significant change to the > structure of the class (adding new methods, changing method or field signatures, > etc) renders serialized forms invalid". It is a common practice to explicitly > assign a serial version UID to a class, thereby disabling this mechanism. > > Classes that expect to evolve over time may use readObject/writeObject and/or > readResolve/writeReplace to customize the mapping between object state and > bytestream. If classes do not use this mechanism, serialization uses a default > schema evolution mechanism to adjust for changes in fields between serialization > and deserialization time; fields that are present in the bytestream but not in > the target class are ignored, and fields that are present in the target class > but not the bytestream get default values (zero, null, etc.) > > Anonymous classes follow the same approach and have access to the same > mechanisms (serialVersionUID, read/writeObject, etc), but they have two > additional sources of instability: > > * The name is generated as EnclosingClass$nnn. Any change to the set of > anonymous classes in the enclosing class may cause sequence numbers to change. > * The number and type of fields (appears in bytecode but not source code) are > generated based on the set of captured values. Any change to the set or > order of captured values can cause these signatures to change (in an > unspecified way). > > If the signatures remain stable, anonymous classes can use serialization hooks > to customize the serialized form, just like named classes. > > The EG has observed that users have largely learned to deal with the problems of > serialization of inner classes, either by (a) don't do it, or (b) ensure that > essentially the same bits are present on both sides of the pipe, preventing skew > from causing instability in either class names or signatures. > > The EG has set, as a minimum bar, that lambda serialization be "at least as good > as" anonymous class serialization. (This is not a high bar.) Further, the EG > has concluded that gratuitous deviations from anonymous class serialization are > undesirable, because, if users have to deal with an imperfect scheme, having > them deal with something that is basically the same as an imperfect scheme > they've already gotten used to is preferable to dealing with a new and > different scheme. > > Further, the EG has rejected the idea of arbitrarily restricting access to > serialization just because it is dangerous; users who have learned to use it > safely should not be unduly encumbered. > > *Failure modes > * > > For anonymous classes, one of two things will happen when attempting to > deserialize after things have changed "too much": > > 1. A deserialization failure due to either the name or signature not matching, > resulting in NoSuchMethodError, IncompatibleClassChangeError, etc. > 2. Deserializing to the wrong thing, without any evidence of error. > > Obviously, a type-2 failure is far worse than a type-1 failure, because no error > is raised and an unintended computation is performed. Here are two examples of > changes that are behaviorally compatible but which will result in type-2 > failures. The first has to do with order-of-declaration. > > *Old code** > * *New code** > * *Result** > * > Runnable r1 = new Runnable() { > void run() { > System.out.println("one"); > } > }; > Runnable r2 = new Runnable() { > void run() { > System.out.println("two"); > } > }; > Runnable r2 = new Runnable() { > void run() { > System.out.println("two"); > } > }; > Runnable r1 = new Runnable() { > void run() { > System.out.println("one"); > } > }; > Deserialized r1 (across skew) prints "two". > > This fails because in both cases, we get classes called Foo$1 and Foo$2, but in > the old code, these correspond to r1 and r2, but in the new code, these > correspond to r2 and r1. > > The other failure has to do with order-of-capture. > > *Old code** > * *New code** > * *Result** > * > String s1 = "foo"; > String s2 = "bar"; > Runnable r = new Runnable() { > void run() { > foo(s1, s2); > } > }; > > String s1 = "foo"; > String s2 = "bar"; > Runnable r = new Runnable() { > void run() { > String s = s2; > foo(s1, s); > } > }; > On deserialization, s1 and s2 are effectively swapped. > > This fails because the order of arguments in the implicitly generated > constructor of the inner class changes due to the order in which the compiler > encounters captured variables. If the reordered variables were of different > types, this would cause a type-1 failure, but if they are the same type, it > causes a type-2 failure. > > *User expectations* > > While experienced users are quick to state the "same bits on both sides" rule > for reliable deserialization, a bit of investigation reveals that user > expectations are actually higher than that. For example, if the compiler > generated a /random/ name for each lambda at compile time, then recompiling the > same source with the same compiler, and using the result for deserialization, > would fail. This is too restrictive; user expectations are not tied to "same > bits", but to a vaguer notion of "I compiled essentially the same source with > essentially the same compiler, and therefore didn't change anything > significant." For example, users would balk if adding a comment or changing > whitespace were to affect deserialization. Users likely expect (in part, due to > behavior of anonymous classes) changes to code that doesn't affect the lambda > directly or indirectly (e.g., add or remove a debugging println) also would not > affect the serialized form. > > In the absence of the user being able to explicitly name the lambda /and/ its > captures (as C++ does), there is no perfect solution. Instead, our goal can > only be to minimize type-2 failures while not unduly creating type-1 failures > when "no significant code change" happened. This means we have to put a stake > in the ground as to what constitutes "significant" code change. > > The de-facto (and likely accidental) definition of "significant" used by inner > classes here is: > > * Adding, removing, or reordering inner class instances earlier in the source > file; > * Changes to the number, order, or type of captured arguments > > This permits changes to code that has nothing to do with inner classes, and many > common refactorings as long as they do not affect the order of inner class > instances or their captures. > > *Current Lambda behavior* > > Lambda serialization currently behaves very similarly to anonymous class > serialization. Where anonymous classes have stable method names but unstable > class names, lambdas are the dual; unstable method names but stable class > names. But since both are used together, the resulting naming stability is > largely the same. > > We do one thing to increase naming stability for lambdas: we hash the name and > signature of the enclosing method in the lambda name. This insulates lambda > naming from the addition, removal, or reordering of methods within a class file, > but naming stability remains sensitive to the order of lambdas within the > method. Similarly, order-of-capture issues are largely similar to inner classes. > > Lambdas bodies are desugared to methods named in the following form: > lambda$/mmm/$/nnn/, where /mmm/ is a hash of the method name and signature, and > /nnn/ is a sequence number of lambdas that have the same /mmm/ hash. > > Because lambdas are instantiated via invokedynamic rather than invoking a > constructor directly, there is also slightly more leniency to changes to the > /types/ of captured argument; changing a captured argument from, say, String to > Object, would be a breaking change for anonymous classes (it changes the > constructor signature) but not for lambdas. This leniency is largely an > accidental artifact of translation, rather than a deliberate design decision. > > *Possible improvements* > > We can start by recognizing the role of the hash of the enclosing method in the > lambda method name. This reduces the set of lambdas that could collide from > "all the lambdas in the file" to "all the lambdas in the method." This reduces > the set of changes that cause both type-1 and type-2 errors. > > An additional observation is that there is a tension between trying to /recover > from/ skew (rather than simply trying to detect it, and failing deserialization) > and complexity. So I think we should focus primarily on detecting skew and > failing deserialization (turning type-2 failures into type-1) while at the same > time not unduly increasing the set of changes that cause type-1 errors, with the > goal of settling on an informal guideline of what constitutes "too much" change. > > We can do this by increasing the number of things that affect the /mmm/ hash, > effectively constructing the lambda-equivalent of the serialization version > UID. The more context we add to this hash, the smaller the set of lambdas that > hash to the same bucket gets, which reduces the space of possible collisions. > The following table shows possible candidates for inclusion, along with examples > of code that illustrate dependence on this item. > > *Item** > * *Old Code** > ------------------------------ > * *New Code** > **------------------------------* > *Effect** > * *Rationale** > * > Names of captured arguments > int x = ... > f(() -> x); > int y = ... > f(() -> y); Including the names of captured arguments in the hash would cause > rename-refactors of captured arguments to be considered a serialization-breaking > change. > While alpha-renaming is generally considered to be semantic-preserving, > serialization has always keyed off of names (such as field names) as being clues > to developer intent. It seems reasonable to say "If you change the names > involved, we have to assume a semantic change occurred." We cannot tell if a > name change is a simple alpha-rename or capturing a completely different > variable, so this is erring on the safe side. > Types of captured arguments > String x = ... > f(() -> x); Object x = ... > f(() -> x); > It seems reasonable to say that, if you capture arguments of a different type, > you've made a semantic change. > Order of captured arguments > () -> { > int a = f(x); > int b = g(y); > return h(a,b); > }; > () -> { > int b = g(y); > int a = f(x); > return h(a,b); > }; Changing the order of capture would become a type-1 failure rather than > possibly a type-2 failure. > Since we cannot detect whether the ordering change is semantically meaningful > or not, it is best to be conservative and say: change to capture order is likely > a semantic change. > Variable assignment target (if present) > Runnable r1 = Foo::f; > Runnable r2 = Foo::g; > Runnable r2 = Foo::g; > Runnable r1 = Foo::f; > > Including variable target name would render this reordering recoverable and correct > If the user has gone to the effort of providing a name, we can use this as a > hint to the meaning of the lambda. > > Runnable r = Foo::f; Runnable runnable = Foo::f; Including variable target > name would render this change (previously recoverable and correct) a > deserialiation failure > If the user has changed the name, it seems reasonable to treat that as possibly > meaning something else. > Target type > Predicate p = String::isEmpty; > Function p = String::isEmpty; Including target type reduces > the space of potential sequence number collisions. > If you've changed the target type, it is a different lambda. > > This list is not exhaustive, and there are others we might consider. (For > example, for lambdas that appear in method invocation context rather than > assignment context, we might include the hash of the invoked method name and > signature, or even the parameter index or name. This is where it starts to > exhibit diminishing returns and increasing brittleness.) > > Taken in total, the effect is: > > * All order-of-capture issues become type-1 failures, rather than type-2 > failures (modulo hash collisions). > * Order of declaration issues are still present, but they are dramatically > reduced, turning many type-2 failures into type-1 failures. > * Some new type-1 failures are introduced, mostly those deriving from > rename-refactors. > > The remaining type-2 failures could be dealt with if we added named lambdas in > the future. (They are also prevented if users always assign lambdas to local > variables whose names are unique within the method; in this way, the > local-variable trick becomes a sort of poor-man's named lambda.) > > We can reduce the probability of collision further by using a different (and > simpler) scheme for non-serializable lambdas (lambda$nnn), so that serializable > lambdas can only accidentally collide with each other. > > However, there are some transformations which we will still not be able to avoid > under this scheme. For example: > > *Old code** > * *New code** > * *Result** > * > Supplier s = > foo ? () -> 1 > : () -> 2; > Supplier s = > !foo ? () -> 2 > : () -> 1; This change is behaviorally compatible but could result in > type-2 failure, since both lambdas have the same target type, capture arity, etc. > > However^2, we can still detect this risk and warn the user. If for any /mmm/, > we issue more than one sequence number /nnn/, we are at risk for a type-2 > failure, and can issue a lint warning in that case, suggesting the user refactor > to something more stable. (Who knows what that diagnostic message will look > like.) With all the hash information above, it seems likely that the number of > potentially colliding lambdas will be small enough that this warning would not > come along too often. > > The impact of this change in the implementation is surprisingly small. It does > not affect the serialized form (java.lang.invoke.SerializedLambda), or the > generated deserialization code ($deserialize$). It only affects the code which > generates the lambda method name, which needs access to a small additional bit > of information -- the assignment target name. Similarly, detecting the > condition required for warning is easy -- "sequence number != 1". > > Qualitatively, the result is still similar in feel to inner classes -- you can > make "irrelevant" changes but we make no heroic attempts to recover from things > like changes in capture order -- but we do a better job of detecting them (and, > if you follow some coding discipline, you can avoid them entirely.) > > From Donald.Raab at gs.com Wed Aug 21 19:29:23 2013 From: Donald.Raab at gs.com (Raab, Donald) Date: Wed, 21 Aug 2013 22:29:23 -0400 Subject: One final stab at improving lambda serialization In-Reply-To: References: <52123BA4.4020700@oracle.com> <52125956.9040200@cs.oswego.edu> Message-ID: <6712820CB52CFB4D842561213A77C05405254F139D@GSCMAMP09EX.firmwide.corp.gs.com> +1 > -----Original Message----- > From: Zakharov, Vladimir [Tech] > Sent: Wednesday, August 21, 2013 10:26 PM > To: 'Doug Lea'; 'Brian Goetz' > Cc: 'lambda-libs-spec-experts at openjdk.java.net'; 'lambda-spec- > experts at openjdk.java.net'; Raab, Donald [Tech] > Subject: RE: One final stab at improving lambda serialization > > I agree. Looks like this proposal strikes just the right balance: it is not being > unnecessarily (and unexpectedly) restrictive while avoiding "silent" failures. I > am supportive of serialization behavior implemented as described below. > > Thank you, > Vlad > > The Goldman Sachs Group, Inc. All rights reserved. > See http://www.gs.com/disclaimer/global_email for important risk > disclosures, conflicts of interest and other terms and conditions relating to > this e-mail and your reliance on information contained in it. This message > may contain confidential or privileged information. If you are not the > intended recipient, please advise us immediately and delete this > message. See http://www.gs.com/disclaimer/email for further information > on confidentiality and the risks of non-secure electronic communication. If > you cannot access these links, please notify us by reply message and we will > send the contents to you. > > > -----Original Message----- > From: lambda-libs-spec-experts-bounces at openjdk.java.net [mailto:lambda- > libs-spec-experts-bounces at openjdk.java.net] On Behalf Of Doug Lea > Sent: Monday, August 19, 2013 1:44 PM > To: Brian Goetz > Cc: lambda-libs-spec-experts at openjdk.java.net; lambda-spec- > experts at openjdk.java.net > Subject: Re: One final stab at improving lambda serialization > > > This is the sort of scheme I had in mind in my reply to David Lloyd (that might > have in part led to this effort). And the details seem reasonable to me. Looks > good to go, assuming no unexpected implementation snags. > > -Doug > > > On 08/19/2013 11:37 AM, Brian Goetz wrote: > > *Background* > > > > The fundamental challenge with serialization is that the code that > > defined a class at serialization time may have changed by the time > > deserialization happens. Serialization is defined to be tolerant of > > change to a certain extent, and admits a degree of customization to allow > additional flexibility. > > > > For ordinary classes, there are three lines of defense: > > > > * serialVersionUID > > * serialization hooks > > * default schema evolution > > > > Serial version UID for the target class must match exactly. By > > default, serialization uses a serial version UID which is a hash of > > the classes signatures. So this default approach means "any > > significant change to the structure of the class (adding new methods, > > changing method or field signatures, > > etc) renders serialized forms invalid". It is a common practice to > > explicitly assign a serial version UID to a class, thereby disabling this > mechanism. > > > > Classes that expect to evolve over time may use readObject/writeObject > > and/or readResolve/writeReplace to customize the mapping between > > object state and bytestream. If classes do not use this mechanism, > > serialization uses a default schema evolution mechanism to adjust for > > changes in fields between serialization and deserialization time; > > fields that are present in the bytestream but not in the target class > > are ignored, and fields that are present in the target class but not > > the bytestream get default values (zero, null, etc.) > > > > Anonymous classes follow the same approach and have access to the same > > mechanisms (serialVersionUID, read/writeObject, etc), but they have > > two additional sources of instability: > > > > * The name is generated as EnclosingClass$nnn. Any change to the set of > > anonymous classes in the enclosing class may cause sequence numbers > to change. > > * The number and type of fields (appears in bytecode but not source > code) are > > generated based on the set of captured values. Any change to the set or > > order of captured values can cause these signatures to change (in an > > unspecified way). > > > > If the signatures remain stable, anonymous classes can use > > serialization hooks to customize the serialized form, just like named classes. > > > > The EG has observed that users have largely learned to deal with the > > problems of serialization of inner classes, either by (a) don't do it, > > or (b) ensure that essentially the same bits are present on both sides > > of the pipe, preventing skew from causing instability in either class names > or signatures. > > > > The EG has set, as a minimum bar, that lambda serialization be "at > > least as good as" anonymous class serialization. (This is not a high > > bar.) Further, the EG has concluded that gratuitous deviations from > > anonymous class serialization are undesirable, because, if users have > > to deal with an imperfect scheme, having them deal with something that > > is basically the same as an imperfect scheme they've already gotten > > used to is preferable to dealing with a new and different scheme. > > > > Further, the EG has rejected the idea of arbitrarily restricting > > access to serialization just because it is dangerous; users who have > > learned to use it safely should not be unduly encumbered. > > > > *Failure modes > > * > > > > For anonymous classes, one of two things will happen when attempting > > to deserialize after things have changed "too much": > > > > 1. A deserialization failure due to either the name or signature not > matching, > > resulting in NoSuchMethodError, IncompatibleClassChangeError, etc. > > 2. Deserializing to the wrong thing, without any evidence of error. > > > > Obviously, a type-2 failure is far worse than a type-1 failure, > > because no error is raised and an unintended computation is performed. > > Here are two examples of changes that are behaviorally compatible but > > which will result in type-2 failures. The first has to do with order-of- > declaration. > > > > *Old code** > > * *New code** > > * *Result** > > * > > Runnable r1 = new Runnable() { > > void run() { > > System.out.println("one"); > > } > > }; > > Runnable r2 = new Runnable() { > > void run() { > > System.out.println("two"); > > } > > }; > > Runnable r2 = new Runnable() { > > void run() { > > System.out.println("two"); > > } > > }; > > Runnable r1 = new Runnable() { > > void run() { > > System.out.println("one"); > > } > > }; > > Deserialized r1 (across skew) prints "two". > > > > This fails because in both cases, we get classes called Foo$1 and > > Foo$2, but in the old code, these correspond to r1 and r2, but in the > > new code, these correspond to r2 and r1. > > > > The other failure has to do with order-of-capture. > > > > *Old code** > > * *New code** > > * *Result** > > * > > String s1 = "foo"; > > String s2 = "bar"; > > Runnable r = new Runnable() { > > void run() { > > foo(s1, s2); > > } > > }; > > > > String s1 = "foo"; > > String s2 = "bar"; > > Runnable r = new Runnable() { > > void run() { > > String s = s2; > > foo(s1, s); > > } > > }; > > On deserialization, s1 and s2 are effectively swapped. > > > > This fails because the order of arguments in the implicitly generated > > constructor of the inner class changes due to the order in which the > > compiler encounters captured variables. If the reordered variables > > were of different types, this would cause a type-1 failure, but if > > they are the same type, it causes a type-2 failure. > > > > *User expectations* > > > > While experienced users are quick to state the "same bits on both > > sides" rule for reliable deserialization, a bit of investigation > > reveals that user expectations are actually higher than that. For > > example, if the compiler generated a /random/ name for each lambda at > > compile time, then recompiling the same source with the same compiler, > > and using the result for deserialization, would fail. This is too > > restrictive; user expectations are not tied to "same bits", but to a > > vaguer notion of "I compiled essentially the same source with > > essentially the same compiler, and therefore didn't change anything > > significant." For example, users would balk if adding a comment or > > changing whitespace were to affect deserialization. Users likely > > expect (in part, due to behavior of anonymous classes) changes to code > > that doesn't affect the lambda directly or indirectly (e.g., add or remove a > debugging println) also would not affect the serialized form. > > > > In the absence of the user being able to explicitly name the lambda > > /and/ its captures (as C++ does), there is no perfect solution. > > Instead, our goal can only be to minimize type-2 failures while not > > unduly creating type-1 failures when "no significant code change" > > happened. This means we have to put a stake in the ground as to what > constitutes "significant" code change. > > > > The de-facto (and likely accidental) definition of "significant" used > > by inner classes here is: > > > > * Adding, removing, or reordering inner class instances earlier in the > source > > file; > > * Changes to the number, order, or type of captured arguments > > > > This permits changes to code that has nothing to do with inner > > classes, and many common refactorings as long as they do not affect > > the order of inner class instances or their captures. > > > > *Current Lambda behavior* > > > > Lambda serialization currently behaves very similarly to anonymous > > class serialization. Where anonymous classes have stable method names > > but unstable class names, lambdas are the dual; unstable method names > > but stable class names. But since both are used together, the > > resulting naming stability is largely the same. > > > > We do one thing to increase naming stability for lambdas: we hash the > > name and signature of the enclosing method in the lambda name. This > > insulates lambda naming from the addition, removal, or reordering of > > methods within a class file, but naming stability remains sensitive to > > the order of lambdas within the method. Similarly, order-of-capture issues > are largely similar to inner classes. > > > > Lambdas bodies are desugared to methods named in the following form: > > lambda$/mmm/$/nnn/, where /mmm/ is a hash of the method name and > > signature, and /nnn/ is a sequence number of lambdas that have the same > /mmm/ hash. > > > > Because lambdas are instantiated via invokedynamic rather than > > invoking a constructor directly, there is also slightly more leniency > > to changes to the /types/ of captured argument; changing a captured > > argument from, say, String to Object, would be a breaking change for > > anonymous classes (it changes the constructor signature) but not for > > lambdas. This leniency is largely an accidental artifact of translation, rather > than a deliberate design decision. > > > > *Possible improvements* > > > > We can start by recognizing the role of the hash of the enclosing > > method in the lambda method name. This reduces the set of lambdas > > that could collide from "all the lambdas in the file" to "all the > > lambdas in the method." This reduces the set of changes that cause both > type-1 and type-2 errors. > > > > An additional observation is that there is a tension between trying to > > /recover from/ skew (rather than simply trying to detect it, and > > failing deserialization) and complexity. So I think we should focus > > primarily on detecting skew and failing deserialization (turning > > type-2 failures into type-1) while at the same time not unduly > > increasing the set of changes that cause type-1 errors, with the goal of > settling on an informal guideline of what constitutes "too much" change. > > > > We can do this by increasing the number of things that affect the > > /mmm/ hash, effectively constructing the lambda-equivalent of the > > serialization version UID. The more context we add to this hash, the > > smaller the set of lambdas that hash to the same bucket gets, which > reduces the space of possible collisions. > > The following table shows possible candidates for inclusion, along > > with examples of code that illustrate dependence on this item. > > > > *Item** > > * *Old Code** > > ------------------------------ > > * *New Code** > > **------------------------------* > > *Effect** > > * *Rationale** > > * > > Names of captured arguments > > int x = ... > > f(() -> x); > > int y = ... > > f(() -> y); Including the names of captured arguments in the hash > would cause > > rename-refactors of captured arguments to be considered a > > serialization-breaking change. > > While alpha-renaming is generally considered to be > > semantic-preserving, serialization has always keyed off of names (such > > as field names) as being clues to developer intent. It seems > > reasonable to say "If you change the names involved, we have to assume > > a semantic change occurred." We cannot tell if a name change is a > > simple alpha-rename or capturing a completely different variable, so this is > erring on the safe side. > > Types of captured arguments > > String x = ... > > f(() -> x); Object x = ... > > f(() -> x); > > It seems reasonable to say that, if you capture arguments of a > > different type, you've made a semantic change. > > Order of captured arguments > > () -> { > > int a = f(x); > > int b = g(y); > > return h(a,b); > > }; > > () -> { > > int b = g(y); > > int a = f(x); > > return h(a,b); > > }; Changing the order of capture would become a type-1 failure rather > than > > possibly a type-2 failure. > > Since we cannot detect whether the ordering change is semantically > > meaningful or not, it is best to be conservative and say: change to > > capture order is likely a semantic change. > > Variable assignment target (if present) > > Runnable r1 = Foo::f; > > Runnable r2 = Foo::g; > > Runnable r2 = Foo::g; > > Runnable r1 = Foo::f; > > > > Including variable target name would render this reordering > recoverable and correct > > If the user has gone to the effort of providing a name, we can use > > this as a hint to the meaning of the lambda. > > > > Runnable r = Foo::f; Runnable runnable = Foo::f; Including > variable target > > name would render this change (previously recoverable and correct) a > > deserialiation failure > > If the user has changed the name, it seems reasonable to treat that > > as possibly meaning something else. > > Target type > > Predicate p = String::isEmpty; > > Function p = String::isEmpty; Including target type > reduces > > the space of potential sequence number collisions. > > If you've changed the target type, it is a different lambda. > > > > This list is not exhaustive, and there are others we might consider. > > (For example, for lambdas that appear in method invocation context > > rather than assignment context, we might include the hash of the > > invoked method name and signature, or even the parameter index or > > name. This is where it starts to exhibit diminishing returns and > > increasing brittleness.) > > > > Taken in total, the effect is: > > > > * All order-of-capture issues become type-1 failures, rather than type-2 > > failures (modulo hash collisions). > > * Order of declaration issues are still present, but they are dramatically > > reduced, turning many type-2 failures into type-1 failures. > > * Some new type-1 failures are introduced, mostly those deriving from > > rename-refactors. > > > > The remaining type-2 failures could be dealt with if we added named > > lambdas in the future. (They are also prevented if users always > > assign lambdas to local variables whose names are unique within the > > method; in this way, the local-variable trick becomes a sort of > > poor-man's named lambda.) > > > > We can reduce the probability of collision further by using a > > different (and > > simpler) scheme for non-serializable lambdas (lambda$nnn), so that > > serializable lambdas can only accidentally collide with each other. > > > > However, there are some transformations which we will still not be > > able to avoid under this scheme. For example: > > > > *Old code** > > * *New code** > > * *Result** > > * > > Supplier s = > > foo ? () -> 1 > > : () -> 2; > > Supplier s = > > !foo ? () -> 2 > > : () -> 1; This change is behaviorally compatible but could result in > > type-2 failure, since both lambdas have the same target type, capture arity, > etc. > > > > However^2, we can still detect this risk and warn the user. If for > > any /mmm/, we issue more than one sequence number /nnn/, we are at > > risk for a type-2 failure, and can issue a lint warning in that case, > > suggesting the user refactor to something more stable. (Who knows > > what that diagnostic message will look > > like.) With all the hash information above, it seems likely that the > > number of potentially colliding lambdas will be small enough that this > > warning would not come along too often. > > > > The impact of this change in the implementation is surprisingly small. > > It does not affect the serialized form > > (java.lang.invoke.SerializedLambda), or the generated deserialization > > code ($deserialize$). It only affects the code which generates the > > lambda method name, which needs access to a small additional bit of > > information -- the assignment target name. Similarly, detecting the > condition required for warning is easy -- "sequence number != 1". > > > > Qualitatively, the result is still similar in feel to inner classes -- > > you can make "irrelevant" changes but we make no heroic attempts to > > recover from things like changes in capture order -- but we do a > > better job of detecting them (and, if you follow some coding > > discipline, you can avoid them entirely.) > > > > From brian.goetz at oracle.com Thu Aug 22 09:29:51 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 22 Aug 2013 12:29:51 -0400 Subject: Milestone: zero open API issues Message-ID: <52163C7F.3040006@oracle.com> We have reached the milestone of "zero open API issues" for Lambda Libraries! The last two open issues -- comparator combinators, AutoCloseable -- are now entering the review process. I still need the help of the EG to review (and contribute to!) the specs for java.util.stream. Candidate docs have been up for a month at: http://cr.openjdk.java.net/~briangoetz/doctmp/doc/ Please, try to review at least the Javadoc for Stream and the package doc. We all know how this API is supposed to work; what we need to do now is make sure that our documentation conveys that spirit, and effective examples, for helping people "get it". From brian.goetz at oracle.com Wed Aug 28 14:58:01 2013 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 28 Aug 2013 17:58:01 -0400 Subject: Specs Message-ID: <521E7269.6010502@oracle.com> Have I mentioned that reviews of the specs and package doc at: http://cr.openjdk.java.net/~briangoetz/doctmp/doc/ would be appreciated?