From github.com+1701815+mkarg at openjdk.java.net Sun Aug 1 07:51:37 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 1 Aug 2021 07:51:37 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v11] In-Reply-To: References: Message-ID: > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has updated the pull request incrementally with one additional commit since the last revision: [Draft] Using blocking lock to prevent concurrently switching into non-blocking mode ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/8e2889fd..f4485d5b Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=10 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=09-10 Stats: 16 lines in 1 file changed: 8 ins; 0 del; 8 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 1 07:51:37 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 1 Aug 2021 07:51:37 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Sat, 31 Jul 2021 21:20:28 GMT, Markus KARG wrote: > I suspect the eventually patch will need have to make use of the blockingLock to prevent the underlying channels from being changed to non-blocking during the transfer. The blocking lock now is used since https://github.com/openjdk/jdk/pull/4263/commits/f4485d5b6a3b5c471feff5642dfef0fc747d3fac to prevent this. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From alanb at openjdk.java.net Sun Aug 1 08:00:47 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Sun, 1 Aug 2021 08:00:47 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call In-Reply-To: References: Message-ID: On Thu, 29 Jul 2021 23:08:58 GMT, Brian Burkhalter wrote: > Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. Changes requested by alanb (Reviewer). src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 705: > 703: break; > 704: position += n; > 705: bytesTransferred += n; I think this patch is problematic for the cases where the second (or subsequent) call to transferTo fails transferToArbitraryChannel has to deal with this by returning the number of bytes transferred before the error. I think it might be better to just continue with having transferTo map to one syscall where possible. In this case I'd prefer not introduce a loop but instead just use the OS "max transfer size" to cap the number of bytes transferred in one call to this method. ------------- PR: https://git.openjdk.java.net/jdk/pull/4940 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 1 08:02:48 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 1 Aug 2021 08:02:48 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Sun, 1 Aug 2021 07:46:36 GMT, Markus KARG wrote: >>> I need to look at it closely but I suspect this introduces a potential overflow. Also if output stream is backed by a SocketChannel configured non-blocking then FC::transferTo may return 0 so I assume there is a potential infinite loop there too. I suspect the eventually patch will need have to make use of the blockingLock to prevent the underlying channels from being changed to non-blocking during the transfer. >> >> I need to confess that my NIO knowledge is not deep enough to follow you closely, so I trust on your advice how to go on from here. > >> I suspect the eventually patch will need have to make use of the blockingLock to prevent the underlying channels from being changed to non-blocking during the transfer. > > The blocking lock now is used since https://github.com/openjdk/jdk/pull/4263/commits/f4485d5b6a3b5c471feff5642dfef0fc747d3fac to prevent this. > Also can you can check that IllegalBlockingModeException is thrown for the case ch is a SelectableChannel configured non-blocking and the destination is a FileChannel? Do we really only have to check this in the particular case of `FileChannel` destinations? And don't we have to assert blocking mode for *destination* channels, too (just like `ChannelOutputStream::writeFully` does)? As this results in an 2:2 matrix of possibilities (src is selectable nor not, dst is selectable or not) it might be easier to do *only the blocking checks* in `transferTo` but let it call something like `transferFromSelectableToNonSelectable` in turn *or* to wrap *each* implementation of `transferTo` in a wrapper like `executeInBlockingMode((src, dst) -> transferTo(src, dst))`...? ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 1 08:27:42 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 1 Aug 2021 08:27:42 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Sat, 31 Jul 2021 17:33:50 GMT, Alan Bateman wrote: >>> Also can you can check that IllegalBlockingModeException is thrown for the case ch is a SelectableChannel configured non-blocking and the destination is a FileChannel? >> >> Done in https://github.com/openjdk/jdk/pull/4263/commits/8e2889fd6138d8aa8e308a1cd2aea3546a7c43a7, but honestly I'd kindly like to ask for a brief explanation why this has to be done. What actual bad effect would happen if I do not throw? > >> The modified code found in [4b501b2](https://github.com/openjdk/jdk/commit/4b501b205c6f1c48bbc82d7a154aed3fc41b1a20) should be safe from infinite loops, as it checks the actual file length in each iteration (even if this costs us one more `synchronized` per iteration). > > I need to look at it closely but I suspect this introduces a potential overflow. Also if output stream is backed by a SocketChannel configured non-blocking then FC::transferTo may return 0 so I assume there is a potential infinite loop there too. I suspect the eventually patch will need have to make use of the blockingLock to prevent the underlying channels from being changed to non-blocking during the transfer. @AlanBateman Did I recap the sum your comments correctly, is the above conclusion what you wanted to tell me? Shall I proceed with one of the two solutions outlined in the "...2:2 matrix..." section of my answer *or* shall I wait until you had a deeper look? ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 1 15:11:44 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 1 Aug 2021 15:11:44 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v12] In-Reply-To: References: Message-ID: > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: [Draft] Using blocking lock to prevent concurrently switching into non-blocking mode ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/f4485d5b..55c96880 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=11 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=10-11 Stats: 16 lines in 1 file changed: 8 ins; 6 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From mik3hall at gmail.com Sun Aug 1 15:17:40 2021 From: mik3hall at gmail.com (Michael Hall) Date: Sun, 1 Aug 2021 10:17:40 -0500 Subject: FileSystemProvider In-Reply-To: References: <526680A4-7424-49B9-AA08-D690D0244126@gmail.com> Message-ID: <554A63A5-5E59-400F-88AC-706FB8D7D03D@gmail.com> > On Apr 30, 2021, at 11:17 AM, Alan Bateman wrote: > > On 29/04/2021 01:13, Michael Hall wrote: >> : >> >> The error is if -Djava.security.manager and -Djava.security.policy are used in invocation?. >> >> > Thanks for the bug report. It's in JBS as JDK-8266345. I think this is another case where the PolicyFile code needs to be very defensive as opening the policy file is a privileged operation. Also interposing on the default file system provider sets up a sandwich where the PolicyFile is going through untrusted code to get to the file system. Getting all these pieces to work together may require the PolicyFile code to use the built-in file system provider directly, needs more investigation. > > -Alan An older one. I was just thinking on this though. My own Default provider is pretty much pass through. I think it was required for me to add my own attribute classes. If these attribute classes could be added directly to the existing platform Default I could eliminate the need to override provider. This might eliminate some of these security concerns and startup timing issues. Possibly there might be other security concerns for Default extensions (plugins?). But it would get me out of the startup process. I haven?t double checked yet to see if I could figure out a way to do so now without jdk changes. From Alan.Bateman at oracle.com Sun Aug 1 18:35:25 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 1 Aug 2021 19:35:25 +0100 Subject: FileSystemProvider In-Reply-To: <554A63A5-5E59-400F-88AC-706FB8D7D03D@gmail.com> References: <526680A4-7424-49B9-AA08-D690D0244126@gmail.com> <554A63A5-5E59-400F-88AC-706FB8D7D03D@gmail.com> Message-ID: On 01/08/2021 16:17, Michael Hall wrote: > : > An older one. I was just thinking on this though. My own Default provider is pretty much pass through. I think it was required for me to add my own attribute classes. If these attribute classes could be added directly to the existing platform Default I could eliminate the need to override provider. This might eliminate some of these security concerns and startup timing issues. Possibly there might be other security concerns for Default extensions (plugins?). But it would get me out of the startup process. > I haven?t double checked yet to see if I could figure out a way to do so now without jdk changes. I think you mentioned in a mail a long time ago that the motivation was creator codes. As I understand it, these were deprecated more than 10 years ago in macOS 10.6. That said, if they are still relevant then I assume they are maintained as extended attributes, meaning using the UserDefinedAttributeView may provide to get to them if the format is documented. -Alan From mik3hall at gmail.com Sun Aug 1 18:52:50 2021 From: mik3hall at gmail.com (Michael Hall) Date: Sun, 1 Aug 2021 13:52:50 -0500 Subject: FileSystemProvider In-Reply-To: References: <526680A4-7424-49B9-AA08-D690D0244126@gmail.com> <554A63A5-5E59-400F-88AC-706FB8D7D03D@gmail.com> Message-ID: <2E0CE205-D547-41FB-9E93-A55E34994946@gmail.com> > On Aug 1, 2021, at 1:35 PM, Alan Bateman wrote: > > On 01/08/2021 16:17, Michael Hall wrote: >> : >> An older one. I was just thinking on this though. My own Default provider is pretty much pass through. I think it was required for me to add my own attribute classes. If these attribute classes could be added directly to the existing platform Default I could eliminate the need to override provider. This might eliminate some of these security concerns and startup timing issues. Possibly there might be other security concerns for Default extensions (plugins?). But it would get me out of the startup process. >> I haven?t double checked yet to see if I could figure out a way to do so now without jdk changes. > I think you mentioned in a mail a long time ago that the motivation was creator codes. As I understand it, these were deprecated more than 10 years ago in macOS 10.6. That said, if they are still relevant then I assume they are maintained as extended attributes, meaning using the UserDefinedAttributeView may provide to get to them if the format is documented. > > -Alan Creator codes are included in a couple different API?s that I provide attributes for. There is a fair amount of redundancy. But I provide attributes for Cocoa, Launch Services, Extended Attributes and Finder file API?s for OS/X. I don?t know if it provides anything essential that Java doesn?t already provide. But it does demonstrate a mechanism through the nio API?s for developers to get to native API?s that Java has not opted to provide. For example Launch Services can provide the default application associated with a file or a list of Applications that can be used. Someone might find a use for something there. I think I probably had some extended attribute support before Java provided it. But from the issues I have run into there might be a concern for anyone else in the future attempting to override the default provider. Extending the platform provider in some way might be an easier way to accomplish this having less collision with JDK bootstrapping code. From github.com+1701815+mkarg at openjdk.java.net Sun Aug 1 22:01:33 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 1 Aug 2021 22:01:33 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: Message-ID: > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has updated the pull request incrementally with two additional commits since the last revision: - Draft: Eliminated duplicate code using lambda expressions - Draft: Use blocking mode also for target channel ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/55c96880..4a0d7cf7 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=12 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=11-12 Stats: 43 lines in 1 file changed: 21 ins; 8 del; 14 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 1 22:01:33 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 1 Aug 2021 22:01:33 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Sun, 1 Aug 2021 08:24:13 GMT, Markus KARG wrote: >>> The modified code found in [4b501b2](https://github.com/openjdk/jdk/commit/4b501b205c6f1c48bbc82d7a154aed3fc41b1a20) should be safe from infinite loops, as it checks the actual file length in each iteration (even if this costs us one more `synchronized` per iteration). >> >> I need to look at it closely but I suspect this introduces a potential overflow. Also if output stream is backed by a SocketChannel configured non-blocking then FC::transferTo may return 0 so I assume there is a potential infinite loop there too. I suspect the eventually patch will need have to make use of the blockingLock to prevent the underlying channels from being changed to non-blocking during the transfer. > > @AlanBateman Did I recap the sum your comments correctly, is the above conclusion what you wanted to tell me? Shall I proceed with one of the two solutions outlined in the "...2:2 matrix..." section of my answer *or* shall I wait until you had a deeper look? Asserting blocking mode for *both* sides (source and target) in https://github.com/openjdk/jdk/pull/4263/commits/fc38eae44de9e16468d33bd2ebab6502c92b4860. Eliminated the resulting duplicate code in https://github.com/openjdk/jdk/pull/4263/commits/4a0d7cf74ee7e35aa0448df4b5ea4c5e3113ece6. Do you see more problems we need to solve? ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From redestad at openjdk.java.net Mon Aug 2 11:40:36 2021 From: redestad at openjdk.java.net (Claes Redestad) Date: Mon, 2 Aug 2021 11:40:36 GMT Subject: RFR: 8263561: Re-examine uses of LinkedList [v5] In-Reply-To: References: Message-ID: <_IT0JRHdfpmkeZIKV0F3138xY_AJgeRi8_-iRh0ZgkQ=.2b1bfed1-0473-4f45-be5a-8da3a3b14406@github.com> On Mon, 26 Jul 2021 08:27:20 GMT, ?????? ??????? wrote: >> After I've renamed remove branch GitHub for some reason has closed original https://github.com/openjdk/jdk/pull/2744, so I've decided to recreate it. > > ?????? ??????? has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 10 commits: > > - Merge branch 'master' into 8263561 > - Merge branch 'master' into 8263561 > - Merge branch 'master' into 8263561 > - Merge branch 'master' into 8263561 > - Merge branch 'master' into 8263561 > > # Conflicts: > # src/java.base/unix/classes/sun/net/dns/ResolverConfigurationImpl.java > - Merge branch 'master' into purge-linked-list > - 8263561: Use sized constructor where reasonable > - 8263561: Use interface List instead of particular type where possible > - 8263561: Rename requestList -> requests > - 8263561: Re-examine uses of LinkedList I always approve of removing LinkedLists and Vectors. Using ArrayDeque in AbstractPoller seems like the right choice. ------------- Marked as reviewed by redestad (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/4304 From github.com+10835776+stsypanov at openjdk.java.net Mon Aug 2 12:39:47 2021 From: github.com+10835776+stsypanov at openjdk.java.net (=?UTF-8?B?0KHQtdGA0LPQtdC5?= =?UTF-8?B?IA==?= =?UTF-8?B?0KbRi9C/0LDQvdC+0LI=?=) Date: Mon, 2 Aug 2021 12:39:47 GMT Subject: RFR: 8263561: Re-examine uses of LinkedList [v6] In-Reply-To: References: Message-ID: > After I've renamed remove branch GitHub for some reason has closed original https://github.com/openjdk/jdk/pull/2744, so I've decided to recreate it. ?????? ??????? has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 11 commits: - Merge branch 'master' into 8263561 # Conflicts: # src/java.base/share/classes/jdk/internal/util/jar/JarIndex.java - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 # Conflicts: # src/java.base/unix/classes/sun/net/dns/ResolverConfigurationImpl.java - Merge branch 'master' into purge-linked-list - 8263561: Use sized constructor where reasonable - 8263561: Use interface List instead of particular type where possible - 8263561: Rename requestList -> requests - ... and 1 more: https://git.openjdk.java.net/jdk/compare/7cc1eb3e...dea42cac ------------- Changes: https://git.openjdk.java.net/jdk/pull/4304/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=4304&range=05 Stats: 47 lines in 9 files changed: 0 ins; 2 del; 45 mod Patch: https://git.openjdk.java.net/jdk/pull/4304.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4304/head:pull/4304 PR: https://git.openjdk.java.net/jdk/pull/4304 From github.com+10835776+stsypanov at openjdk.java.net Mon Aug 2 12:53:37 2021 From: github.com+10835776+stsypanov at openjdk.java.net (=?UTF-8?B?0KHQtdGA0LPQtdC5?= =?UTF-8?B?IA==?= =?UTF-8?B?0KbRi9C/0LDQvdC+0LI=?=) Date: Mon, 2 Aug 2021 12:53:37 GMT Subject: Integrated: 8263561: Re-examine uses of LinkedList In-Reply-To: References: Message-ID: On Wed, 2 Jun 2021 12:10:46 GMT, ?????? ??????? wrote: > After I've renamed remove branch GitHub for some reason has closed original https://github.com/openjdk/jdk/pull/2744, so I've decided to recreate it. This pull request has now been integrated. Changeset: 249d6418 Author: Sergey Tsypanov Committer: Claes Redestad URL: https://git.openjdk.java.net/jdk/commit/249d641889c6f9aed6957502d5fca9c74c9baceb Stats: 47 lines in 9 files changed: 0 ins; 2 del; 45 mod 8263561: Re-examine uses of LinkedList Reviewed-by: redestad ------------- PR: https://git.openjdk.java.net/jdk/pull/4304 From lancea at openjdk.java.net Mon Aug 2 15:50:51 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Mon, 2 Aug 2021 15:50:51 GMT Subject: Integrated: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside In-Reply-To: References: Message-ID: On Sun, 25 Jul 2021 21:56:10 GMT, Lance Andersen wrote: > Hi, > > As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS > > Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. > > > Mach5 tiers 1 through 3 have been run without any errors encountered . > > Best, > Lance This pull request has now been integrated. Changeset: 3e3051e2 Author: Lance Andersen URL: https://git.openjdk.java.net/jdk/commit/3e3051e2ee93142983e9a3edee038e4f7b5ac0f2 Stats: 201 lines in 3 files changed: 200 ins; 0 del; 1 mod 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside Reviewed-by: alanb, naoto ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From bpb at openjdk.java.net Mon Aug 2 21:58:16 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Mon, 2 Aug 2021 21:58:16 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call [v2] In-Reply-To: References: Message-ID: > Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8271308: Make direct transfer map to only one syscall ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4940/files - new: https://git.openjdk.java.net/jdk/pull/4940/files/3f7836f0..86e2d65d Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4940&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4940&range=00-01 Stats: 57 lines in 4 files changed: 13 ins; 18 del; 26 mod Patch: https://git.openjdk.java.net/jdk/pull/4940.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4940/head:pull/4940 PR: https://git.openjdk.java.net/jdk/pull/4940 From bpb at openjdk.java.net Mon Aug 2 21:58:19 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Mon, 2 Aug 2021 21:58:19 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call [v2] In-Reply-To: References: Message-ID: On Sun, 1 Aug 2021 07:57:15 GMT, Alan Bateman wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8271308: Make direct transfer map to only one syscall > > src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 705: > >> 703: break; >> 704: position += n; >> 705: bytesTransferred += n; > > I think this patch is problematic for the cases where the second (or subsequent) call to transferTo fails. transferToArbitraryChannel has to deal with this by returning the number of bytes transferred before the error. > > I think it might be better to just continue with having transferTo map to one syscall where possible. In this case I'd prefer not introduce a loop but instead just use the OS "max transfer size" to cap the number of bytes transferred in one call to this method. In the updated code, the direct transfer case no longer loops and just maps to one syscall. The previously introduced private `transferTo()` method is removed. `transferToArbitraryChannel()` is modified to have a `long`-valued count. `transferToTrustedChannel()` works out of the box as-is and is not changed. ------------- PR: https://git.openjdk.java.net/jdk/pull/4940 From alanb at openjdk.java.net Tue Aug 3 09:45:30 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Tue, 3 Aug 2021 09:45:30 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call [v2] In-Reply-To: References: Message-ID: <2IoZfj8EyaKv2omc2V_K17ast4yge4g37lTs9ooN-No=.eb1e63fa-f17c-4e8f-83bf-c80cc46f1dde@github.com> On Mon, 2 Aug 2021 21:54:10 GMT, Brian Burkhalter wrote: >> src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 705: >> >>> 703: break; >>> 704: position += n; >>> 705: bytesTransferred += n; >> >> I think this patch is problematic for the cases where the second (or subsequent) call to transferTo fails. transferToArbitraryChannel has to deal with this by returning the number of bytes transferred before the error. >> >> I think it might be better to just continue with having transferTo map to one syscall where possible. In this case I'd prefer not introduce a loop but instead just use the OS "max transfer size" to cap the number of bytes transferred in one call to this method. > > In the updated code, the direct transfer case no longer loops and just maps to one syscall. The previously introduced private `transferTo()` method is removed. `transferToArbitraryChannel()` is modified to have a `long`-valued count. `transferToTrustedChannel()` works out of the box as-is and is not changed. The update mostly looks okay, just wondering if could case maxDirectTransferSize to avoid the JNI call every time. ------------- PR: https://git.openjdk.java.net/jdk/pull/4940 From bpb at openjdk.java.net Tue Aug 3 16:03:31 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 3 Aug 2021 16:03:31 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call [v2] In-Reply-To: <2IoZfj8EyaKv2omc2V_K17ast4yge4g37lTs9ooN-No=.eb1e63fa-f17c-4e8f-83bf-c80cc46f1dde@github.com> References: <2IoZfj8EyaKv2omc2V_K17ast4yge4g37lTs9ooN-No=.eb1e63fa-f17c-4e8f-83bf-c80cc46f1dde@github.com> Message-ID: <-v3JYLCke7f7lMZTrNHOVGZjJvPCAE6LW-mbz1oxI28=.f9c81925-8629-4a51-8531-01f5f9510b4f@github.com> On Tue, 3 Aug 2021 09:42:29 GMT, Alan Bateman wrote: >> In the updated code, the direct transfer case no longer loops and just maps to one syscall. The previously introduced private `transferTo()` method is removed. `transferToArbitraryChannel()` is modified to have a `long`-valued count. `transferToTrustedChannel()` works out of the box as-is and is not changed. > > The update mostly looks okay, just wondering if could case maxDirectTransferSize to avoid the JNI call every time. I think it could be a constant equal to either the Linux or Windows value, not `Integer.MAX_VALUE`, as that would force `TransmiteFile()` to fail for the `MAX_VALUE` case. Or the JNI call could be made just once when setting up. ------------- PR: https://git.openjdk.java.net/jdk/pull/4940 From bpb at openjdk.java.net Tue Aug 3 16:29:57 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 3 Aug 2021 16:29:57 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call [v3] In-Reply-To: References: Message-ID: > Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8271308: Add constant MAX_DIRECT_TRANSFER_SIZE initialized from one native call ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4940/files - new: https://git.openjdk.java.net/jdk/pull/4940/files/86e2d65d..f9a6b265 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4940&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4940&range=01-02 Stats: 5 lines in 1 file changed: 3 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/4940.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4940/head:pull/4940 PR: https://git.openjdk.java.net/jdk/pull/4940 From alanb at openjdk.java.net Tue Aug 3 16:35:31 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Tue, 3 Aug 2021 16:35:31 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call [v3] In-Reply-To: References: Message-ID: On Tue, 3 Aug 2021 16:29:57 GMT, Brian Burkhalter wrote: >> Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8271308: Add constant MAX_DIRECT_TRANSFER_SIZE initialized from one native call Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/4940 From mik3hall at gmail.com Tue Aug 3 21:25:19 2021 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 3 Aug 2021 16:25:19 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> Message-ID: > On Jul 30, 2021, at 9:36 AM, Alan Bateman wrote: > > On 30/07/2021 00:00, Michael Hall wrote: >> Correction non-modular exploded works >> >> java 18 exploded works >> /usr/libexec/java_home -v 18 --exec java -cp .:macnio2 -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider org.test.Test >> WARNING: A command line option has enabled the Security Manager >> WARNING: The Security Manager is deprecated and will be removed in a future release >> us.hall.trz.osx.MacFileSystem at 60e53b93 >> us.hall.trz.osx.MacFileSystemProvider >> >> java 17 exploded works >> /usr/libexec/java_home -v 17 --exec java -cp .:macnio2 -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider org.test.Test >> WARNING: A command line option has enabled the Security Manager >> WARNING: The Security Manager is deprecated and will be removed in a future release >> us.hall.trz.osx.MacFileSystem at 511d50c0 >> us.hall.trz.osx.MacFileSystemProvider >> >> Non-exploded, or jar?d, is probably still https://bugs.openjdk.java.net/browse/JDK-8263940 > Yes, the exploded case should be okay. However, if java.nio.file.spi.DefaultFileSystemProvider specifies a class in a JAR file that is on the class path or module path then you'll run into this issue. Setting the security manager on the command line will trigger this occur at VM startup. > -Alan I had filed an incident report against my application for these errors when I was asked to verify the security manager on command line fix. Having no simple test case the bug report was just closed as unable to reproduce. If https://bugs.openjdk.java.net/browse/JDK-8263940 is assumed to still cover the current situation that?s fine. Otherwise there is no bug report I know of to track the current errors. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vtewari at openjdk.java.net Wed Aug 4 03:19:36 2021 From: vtewari at openjdk.java.net (Vyom Tewari) Date: Wed, 4 Aug 2021 03:19:36 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call [v3] In-Reply-To: References: Message-ID: On Tue, 3 Aug 2021 16:29:57 GMT, Brian Burkhalter wrote: >> Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8271308: Add constant MAX_DIRECT_TRANSFER_SIZE initialized from one native call Marked as reviewed by vtewari (Committer). ------------- PR: https://git.openjdk.java.net/jdk/pull/4940 From Alan.Bateman at oracle.com Wed Aug 4 06:17:23 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 4 Aug 2021 07:17:23 +0100 Subject: New release DefaultFileSystemProviders In-Reply-To: References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> Message-ID: <183e5763-96df-8786-1090-7f6f12df9d78@oracle.com> On 03/08/2021 22:25, Michael Hall wrote: > : > > I had filed an incident report against my application for these errors > when I was asked to verify the security manager on command line fix. > Having no simple test case the bug report was just closed as unable to > reproduce. > If https://bugs.openjdk.java.net/browse/JDK-8263940 > ?is assumed to still > cover the current situation that?s fine. Otherwise there is no bug > report I know of to track the current errors. This is the only outstanding issue that I'm aware of. It requires the java.util.zip.ZipFile implementation to be changed to use the built-in file system provider to read the classes for the custom file system that has been packaged in JAR file and configured to be the default file system provider. That's how the recursive initialization issue arises. -Alan From mik3hall at gmail.com Wed Aug 4 10:25:39 2021 From: mik3hall at gmail.com (Michael Hall) Date: Wed, 4 Aug 2021 05:25:39 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: <183e5763-96df-8786-1090-7f6f12df9d78@oracle.com> References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> <183e5763-96df-8786-1090-7f6f12df9d78@oracle.com> Message-ID: <8B71C4A5-73A3-45E7-BA08-C34C48DB5E34@gmail.com> > On Aug 4, 2021, at 1:17 AM, Alan Bateman wrote: > > On 03/08/2021 22:25, Michael Hall wrote: >> : >> >> I had filed an incident report against my application for these errors when I was asked to verify the security manager on command line fix. >> Having no simple test case the bug report was just closed as unable to reproduce. >> If https://bugs.openjdk.java.net/browse/JDK-8263940 is assumed to still cover the current situation that?s fine. Otherwise there is no bug report I know of to track the current errors. > This is the only outstanding issue that I'm aware of. It requires the java.util.zip.ZipFile implementation to be changed to use the built-in file system provider to read the classes for the custom file system that has been packaged in JAR file and configured to be the default file system provider. That's how the recursive initialization issue arises. > > -Alan I?ll wait until that is resolved to see if that is the root cause for the issues I see now. They seem to get to there differently but might end up failing the same. From bpb at openjdk.java.net Wed Aug 4 17:14:00 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 4 Aug 2021 17:14:00 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call [v4] In-Reply-To: References: Message-ID: <726hNSFPTvwzJDWyAgBGV6YD5E2skCX7NkfCK_pmHJc=.1405e322-80ab-4e55-a1cc-1edb7e9591b5@github.com> > Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8271308: Fix static initialization order problem ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4940/files - new: https://git.openjdk.java.net/jdk/pull/4940/files/f9a6b265..631aece4 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4940&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4940&range=02-03 Stats: 2 lines in 1 file changed: 1 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/4940.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4940/head:pull/4940 PR: https://git.openjdk.java.net/jdk/pull/4940 From alanb at openjdk.java.net Wed Aug 4 17:14:04 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Wed, 4 Aug 2021 17:14:04 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call [v4] In-Reply-To: <726hNSFPTvwzJDWyAgBGV6YD5E2skCX7NkfCK_pmHJc=.1405e322-80ab-4e55-a1cc-1edb7e9591b5@github.com> References: <726hNSFPTvwzJDWyAgBGV6YD5E2skCX7NkfCK_pmHJc=.1405e322-80ab-4e55-a1cc-1edb7e9591b5@github.com> Message-ID: On Wed, 4 Aug 2021 17:10:47 GMT, Brian Burkhalter wrote: >> Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8271308: Fix static initialization order problem Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/4940 From vtewari at openjdk.java.net Thu Aug 5 06:21:29 2021 From: vtewari at openjdk.java.net (Vyom Tewari) Date: Thu, 5 Aug 2021 06:21:29 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call [v4] In-Reply-To: <726hNSFPTvwzJDWyAgBGV6YD5E2skCX7NkfCK_pmHJc=.1405e322-80ab-4e55-a1cc-1edb7e9591b5@github.com> References: <726hNSFPTvwzJDWyAgBGV6YD5E2skCX7NkfCK_pmHJc=.1405e322-80ab-4e55-a1cc-1edb7e9591b5@github.com> Message-ID: On Wed, 4 Aug 2021 17:14:00 GMT, Brian Burkhalter wrote: >> Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8271308: Fix static initialization order problem Looks good, much better than calling "maxTransferSize0()" to in "transferTo" method. ------------- Marked as reviewed by vtewari (Committer). PR: https://git.openjdk.java.net/jdk/pull/4940 From bpb at openjdk.java.net Thu Aug 5 16:12:39 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 5 Aug 2021 16:12:39 GMT Subject: Integrated: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call In-Reply-To: References: Message-ID: On Thu, 29 Jul 2021 23:08:58 GMT, Brian Burkhalter wrote: > Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. This pull request has now been integrated. Changeset: e2c5bfe0 Author: Brian Burkhalter URL: https://git.openjdk.java.net/jdk/commit/e2c5bfe083adb82c0d99ba3cb05a9d5cdb8a05cc Stats: 186 lines in 4 files changed: 170 ins; 3 del; 13 mod 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call Reviewed-by: alanb, vtewari ------------- PR: https://git.openjdk.java.net/jdk/pull/4940 From bpb at openjdk.java.net Thu Aug 5 21:47:44 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 5 Aug 2021 21:47:44 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 Message-ID: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. ------------- Commit messages: - 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 Changes: https://git.openjdk.java.net/jdk/pull/5025/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5025&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8272047 Stats: 14 lines in 1 file changed: 6 ins; 0 del; 8 mod Patch: https://git.openjdk.java.net/jdk/pull/5025.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5025/head:pull/5025 PR: https://git.openjdk.java.net/jdk/pull/5025 From bpb at openjdk.java.net Thu Aug 5 21:47:44 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 5 Aug 2021 21:47:44 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 In-Reply-To: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: On Thu, 5 Aug 2021 21:40:43 GMT, Brian Burkhalter wrote: > Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. This change has been verified on several machines of the pertinent platform. ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From dcubed at openjdk.java.net Thu Aug 5 21:47:45 2021 From: dcubed at openjdk.java.net (Daniel D.Daugherty) Date: Thu, 5 Aug 2021 21:47:45 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 In-Reply-To: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: On Thu, 5 Aug 2021 21:40:43 GMT, Brian Burkhalter wrote: > Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. Just a clarification. The test is failing on linux-aarch64 (and not macos-aarch64). ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From naoto at openjdk.java.net Thu Aug 5 23:04:28 2021 From: naoto at openjdk.java.net (Naoto Sato) Date: Thu, 5 Aug 2021 23:04:28 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 In-Reply-To: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: On Thu, 5 Aug 2021 21:40:43 GMT, Brian Burkhalter wrote: > Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. test/jdk/java/nio/channels/FileChannel/Transfer2GPlus.java line 100: > 98: break; > 99: total += n; > 100: } while (total < LENGTH); Could it be possible where the `total` is larger than `LINUX_MAX_TRANSFER_SIZE` here after the loop? If so, the test below only checks whether the `dstCh` is smaller than `LENGTH` and proceeds without the error reported. ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From bpb at openjdk.java.net Thu Aug 5 23:12:40 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 5 Aug 2021 23:12:40 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 In-Reply-To: References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: On Thu, 5 Aug 2021 23:00:58 GMT, Naoto Sato wrote: >> Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. > > test/jdk/java/nio/channels/FileChannel/Transfer2GPlus.java line 100: > >> 98: break; >> 99: total += n; >> 100: } while (total < LENGTH); > > Could it be possible where the `total` is larger than `LINUX_MAX_TRANSFER_SIZE` here after the loop? If so, the test below only checks whether the `dstCh` is smaller than `LENGTH` and proceeds without the error reported. On Linux the `transferTo()` method uses `sendfile()` for the code in question. The specification of [sendfile()](https://man7.org/linux/man-pages/man2/sendfile.2.html) states that it will transfer at most `0x7ffff000` (`LINUX_MAX_TRANSFER_SIZE`) bytes. So here we are only checking for an aberration which is not essential to the test. The important content is when `transferTo()` is supposed to actually copy the full length in one shot. This Linux part of the test is just a drive-by sanity check. ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From naoto at openjdk.java.net Thu Aug 5 23:29:39 2021 From: naoto at openjdk.java.net (Naoto Sato) Date: Thu, 5 Aug 2021 23:29:39 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 In-Reply-To: References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: <2ONwF9eMEuA-1CM0cBKNbpAJUQt6-Vu3VeHlawh5HFM=.08d1a893-1d6e-46f5-a155-0ce1911df2a9@github.com> On Thu, 5 Aug 2021 23:09:27 GMT, Brian Burkhalter wrote: >> test/jdk/java/nio/channels/FileChannel/Transfer2GPlus.java line 100: >> >>> 98: break; >>> 99: total += n; >>> 100: } while (total < LENGTH); >> >> Could it be possible where the `total` is larger than `LINUX_MAX_TRANSFER_SIZE` here after the loop? If so, the test below only checks whether the `dstCh` is smaller than `LENGTH` and proceeds without the error reported. > > On Linux the `transferTo()` method uses `sendfile()` for the code in question. The specification of [sendfile()](https://man7.org/linux/man-pages/man2/sendfile.2.html) states that it will transfer at most `0x7ffff000` (`LINUX_MAX_TRANSFER_SIZE`) bytes. So here we are only checking for an aberration which is not essential to the test. The important content is when `transferTo()` is supposed to actually copy the full length in one shot. This Linux part of the test is just a drive-by sanity check. Thanks for the explanation. Then I would think adding the same check in the loop for consistency, i.e., check the transferred size `n` in one shot is no more than `LINUX_MAX_TRANSFER_SIZE` after line 98. ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From bpb at openjdk.java.net Thu Aug 5 23:33:33 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 5 Aug 2021 23:33:33 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 In-Reply-To: <2ONwF9eMEuA-1CM0cBKNbpAJUQt6-Vu3VeHlawh5HFM=.08d1a893-1d6e-46f5-a155-0ce1911df2a9@github.com> References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> <2ONwF9eMEuA-1CM0cBKNbpAJUQt6-Vu3VeHlawh5HFM=.08d1a893-1d6e-46f5-a155-0ce1911df2a9@github.com> Message-ID: On Thu, 5 Aug 2021 23:26:44 GMT, Naoto Sato wrote: >> On Linux the `transferTo()` method uses `sendfile()` for the code in question. The specification of [sendfile()](https://man7.org/linux/man-pages/man2/sendfile.2.html) states that it will transfer at most `0x7ffff000` (`LINUX_MAX_TRANSFER_SIZE`) bytes. So here we are only checking for an aberration which is not essential to the test. The important content is when `transferTo()` is supposed to actually copy the full length in one shot. This Linux part of the test is just a drive-by sanity check. > > Thanks for the explanation. Then I would think adding the same check in the loop for consistency, i.e., check the transferred size `n` in one shot is no more than `LINUX_MAX_TRANSFER_SIZE` after line 98. I can do that. Thanks. ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From bpb at openjdk.java.net Thu Aug 5 23:55:50 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 5 Aug 2021 23:55:50 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 [v2] In-Reply-To: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: > Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8272047: Update Linux loop in testToFileChannel() ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5025/files - new: https://git.openjdk.java.net/jdk/pull/5025/files/2662e3ac..3ec8212e Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5025&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5025&range=00-01 Stats: 2 lines in 1 file changed: 2 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/5025.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5025/head:pull/5025 PR: https://git.openjdk.java.net/jdk/pull/5025 From bpb at openjdk.java.net Fri Aug 6 00:00:51 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Fri, 6 Aug 2021 00:00:51 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 [v3] In-Reply-To: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: > Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. Brian Burkhalter has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: 8272047: Update Linux loop in testToFileChannel() ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5025/files - new: https://git.openjdk.java.net/jdk/pull/5025/files/3ec8212e..24bd8f0a Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5025&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5025&range=01-02 Stats: 0 lines in 1 file changed: 0 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/5025.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5025/head:pull/5025 PR: https://git.openjdk.java.net/jdk/pull/5025 From naoto at openjdk.java.net Fri Aug 6 00:42:38 2021 From: naoto at openjdk.java.net (Naoto Sato) Date: Fri, 6 Aug 2021 00:42:38 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 [v3] In-Reply-To: References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: <1vyOfEh5m2XONWKuS9f993nrTtoI4jl99Edpd3R8KFE=.3af88fff-a352-40aa-b07e-565b75a9ebcb@github.com> On Fri, 6 Aug 2021 00:00:51 GMT, Brian Burkhalter wrote: >> Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. > > Brian Burkhalter has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. Looks good. Thanks for the fix! ------------- Marked as reviewed by naoto (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/5025 From alanb at openjdk.java.net Fri Aug 6 07:42:30 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 6 Aug 2021 07:42:30 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 [v3] In-Reply-To: References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: On Fri, 6 Aug 2021 00:00:51 GMT, Brian Burkhalter wrote: >> Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. > > Brian Burkhalter has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. As you note, the sendfile(2) page says it is limited 0x7ffff000 but the linux-aarch64 systems where the test is failing is limited to 0x7fff0000. It might better to put this test on the exclude list/ProblemList.txt to allow for more investigation. It might be that the limit returned by FCI.maxDirectTransferSize0 needs to take account of kernel or architecture differences. ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From dcubed at openjdk.java.net Fri Aug 6 13:38:30 2021 From: dcubed at openjdk.java.net (Daniel D.Daugherty) Date: Fri, 6 Aug 2021 13:38:30 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 [v3] In-Reply-To: References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: On Fri, 6 Aug 2021 00:00:51 GMT, Brian Burkhalter wrote: >> Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. > > Brian Burkhalter has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: > > 8272047: Update Linux loop in testToFileChannel() I've created the following sub-task to ProblemList the failing test: JDK-8272095 ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 https://bugs.openjdk.java.net/browse/JDK-8272095 Please update this PR to remove the ProblemListing entry when this fix is integrated. ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From dcubed at openjdk.java.net Fri Aug 6 13:42:44 2021 From: dcubed at openjdk.java.net (Daniel D.Daugherty) Date: Fri, 6 Aug 2021 13:42:44 GMT Subject: Integrated: 8272095: ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 In-Reply-To: References: Message-ID: On Fri, 6 Aug 2021 13:34:37 GMT, Alan Bateman wrote: >> A trivial fix to ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 > > Marked as reviewed by alanb (Reviewer). @AlanBateman - Thanks for the fast review! ------------- PR: https://git.openjdk.java.net/jdk/pull/5032 From alanb at openjdk.java.net Fri Aug 6 13:42:44 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 6 Aug 2021 13:42:44 GMT Subject: Integrated: 8272095: ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 In-Reply-To: References: Message-ID: On Fri, 6 Aug 2021 13:32:00 GMT, Daniel D. Daugherty wrote: > A trivial fix to ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/5032 From dcubed at openjdk.java.net Fri Aug 6 13:42:44 2021 From: dcubed at openjdk.java.net (Daniel D.Daugherty) Date: Fri, 6 Aug 2021 13:42:44 GMT Subject: Integrated: 8272095: ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 Message-ID: A trivial fix to ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 ------------- Commit messages: - 8272095: ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 Changes: https://git.openjdk.java.net/jdk/pull/5032/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5032&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8272095 Stats: 2 lines in 1 file changed: 2 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/5032.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5032/head:pull/5032 PR: https://git.openjdk.java.net/jdk/pull/5032 From dcubed at openjdk.java.net Fri Aug 6 13:42:45 2021 From: dcubed at openjdk.java.net (Daniel D.Daugherty) Date: Fri, 6 Aug 2021 13:42:45 GMT Subject: Integrated: 8272095: ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 In-Reply-To: References: Message-ID: On Fri, 6 Aug 2021 13:32:00 GMT, Daniel D. Daugherty wrote: > A trivial fix to ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 This pull request has now been integrated. Changeset: f4cf2f7c Author: Daniel D. Daugherty URL: https://git.openjdk.java.net/jdk/commit/f4cf2f7cef6c09c16d714f08505327edd7032604 Stats: 2 lines in 1 file changed: 2 ins; 0 del; 0 mod 8272095: ProblemList java/nio/channels/FileChannel/Transfer2GPlus.java on linux-aarch64 Reviewed-by: alanb ------------- PR: https://git.openjdk.java.net/jdk/pull/5032 From bpb at openjdk.java.net Fri Aug 6 15:43:32 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Fri, 6 Aug 2021 15:43:32 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 [v3] In-Reply-To: References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: <9MT7fPEKK8RXqptgqr8KwEHdn0mhwXposLThAVcFjRA=.1be8c995-7d83-44ab-a660-9f771f18ae04@github.com> On Fri, 6 Aug 2021 00:00:51 GMT, Brian Burkhalter wrote: >> Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. > > Brian Burkhalter has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. Or perhaps forget about the Linux limit altogether as that is not the point of the test, which is to verify that more than `Integer.MAX_VALUE` bytes can be transferred in one go. ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From bpb at openjdk.java.net Fri Aug 6 17:04:53 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Fri, 6 Aug 2021 17:04:53 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 [v4] In-Reply-To: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: <5b9T7F25YZMhrc8t8P9FxU0215Vt-CN_ESegNsD2M6Q=.f661d816-c53b-4160-8db6-19239e25a8ad@github.com> > Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8272047: Remove testing against LINUX_MAX_TRANSFER_SIZE on Linux ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5025/files - new: https://git.openjdk.java.net/jdk/pull/5025/files/24bd8f0a..4ea7c9fb Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5025&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5025&range=02-03 Stats: 8 lines in 1 file changed: 0 ins; 5 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/5025.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5025/head:pull/5025 PR: https://git.openjdk.java.net/jdk/pull/5025 From bpb at openjdk.java.net Fri Aug 6 17:28:00 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Fri, 6 Aug 2021 17:28:00 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 [v5] In-Reply-To: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: > Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. Brian Burkhalter has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision: - 8272047: Remove Transfer2GPlus from problem list - Merge - 8272047: Remove testing against LINUX_MAX_TRANSFER_SIZE on Linux - 8272047: Update Linux loop in testToFileChannel() - 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5025/files - new: https://git.openjdk.java.net/jdk/pull/5025/files/4ea7c9fb..0c5b2c94 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5025&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5025&range=03-04 Stats: 1528 lines in 84 files changed: 985 ins; 258 del; 285 mod Patch: https://git.openjdk.java.net/jdk/pull/5025.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5025/head:pull/5025 PR: https://git.openjdk.java.net/jdk/pull/5025 From alanb at openjdk.java.net Sat Aug 7 05:49:34 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Sat, 7 Aug 2021 05:49:34 GMT Subject: RFR: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 [v5] In-Reply-To: References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: On Fri, 6 Aug 2021 17:28:00 GMT, Brian Burkhalter wrote: >> Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. > > Brian Burkhalter has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision: > > - 8272047: Remove Transfer2GPlus from problem list > - Merge > - 8272047: Remove testing against LINUX_MAX_TRANSFER_SIZE on Linux > - 8272047: Update Linux loop in testToFileChannel() > - 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From github.com+1701815+mkarg at openjdk.java.net Sat Aug 7 06:48:34 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 7 Aug 2021 06:48:34 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: Message-ID: <5v2JxcXvfhlsA_lPDTS1u7LQul5S12tNrFHtN3lmnK8=.0186f02f-6537-4ae6-beda-a4268bc047a9@github.com> On Sun, 1 Aug 2021 22:01:33 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has updated the pull request incrementally with two additional commits since the last revision: > > - Draft: Eliminated duplicate code using lambda expressions > - Draft: Use blocking mode also for target channel I think I fixed all requested changes. Anymore comments on this PR? ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From bpb at openjdk.java.net Mon Aug 9 16:54:39 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Mon, 9 Aug 2021 16:54:39 GMT Subject: Integrated: 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 In-Reply-To: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> References: <0SnbeJlcuJVEyIhpEAdC2VDFd2jEthCytZqUQJGOc08=.f08297ed-41c4-404b-b20c-739cb7e2ad7d@github.com> Message-ID: <_uP3PY1m8Ldb8zP_eVU0-3xbw2utDiW4mjZ7HfX_Trw=.c2aed022-dda3-49a5-9add-7ab50bb8ac9b@github.com> On Thu, 5 Aug 2021 21:40:43 GMT, Brian Burkhalter wrote: > Please consider this fix which resolves a test consistently failing on aarch64 in the CI pipeline. This pull request has now been integrated. Changeset: b53828b7 Author: Brian Burkhalter URL: https://git.openjdk.java.net/jdk/commit/b53828b7c219f8f8fb22177bcc80b045ef7bad76 Stats: 17 lines in 2 files changed: 5 ins; 4 del; 8 mod 8272047: java/nio/channels/FileChannel/Transfer2GPlus.java failed with Unexpected transfer size: 2147418112 Reviewed-by: naoto, alanb ------------- PR: https://git.openjdk.java.net/jdk/pull/5025 From alanb at openjdk.java.net Mon Aug 9 18:13:40 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Mon, 9 Aug 2021 18:13:40 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: <5v2JxcXvfhlsA_lPDTS1u7LQul5S12tNrFHtN3lmnK8=.0186f02f-6537-4ae6-beda-a4268bc047a9@github.com> References: <5v2JxcXvfhlsA_lPDTS1u7LQul5S12tNrFHtN3lmnK8=.0186f02f-6537-4ae6-beda-a4268bc047a9@github.com> Message-ID: On Sat, 7 Aug 2021 06:45:21 GMT, Markus KARG wrote: > I think I fixed all requested changes. Anymore comments on this PR? I hope to get to this soon. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From bpb at openjdk.java.net Tue Aug 10 01:28:59 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 10 Aug 2021 01:28:59 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap Message-ID: This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. ------------- Commit messages: - 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap Changes: https://git.openjdk.java.net/jdk/pull/5061/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5061&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8140241 Stats: 163 lines in 6 files changed: 157 ins; 0 del; 6 mod Patch: https://git.openjdk.java.net/jdk/pull/5061.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5061/head:pull/5061 PR: https://git.openjdk.java.net/jdk/pull/5061 From bpb at openjdk.java.net Tue Aug 10 01:28:59 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 10 Aug 2021 01:28:59 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap In-Reply-To: References: Message-ID: On Tue, 10 Aug 2021 01:21:21 GMT, Brian Burkhalter wrote: > This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. This change would work around a bug specific to macOS which the operating system vendor has not fixed in more than five years. The JDK issue was filed in 2015 after having been observed on macOS 10.5.5 (Yosemite); the bug has been observed on macOS versions through 10.15.7 (Catalina). When run on macOS without the source fix, the included test hangs and its process cannot in general be killed without restarting or in some cases power cycling the machine. ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From alanb at openjdk.java.net Tue Aug 10 09:09:30 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Tue, 10 Aug 2021 09:09:30 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap In-Reply-To: References: Message-ID: <9f2nJSvDOoYQqszAHY6MGJNYUq0ohqCBqPk2oSQTlJI=.d8b6403b-3d59-44ea-a5db-aae3f0582307@github.com> On Tue, 10 Aug 2021 01:21:21 GMT, Brian Burkhalter wrote: > This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. This is logged with Apple as 37764443, it would be useful if we could get an update on that. I'm not opposed to putting a workaround for this macOS bug but it does add clutter for scenario that may not arise too often. If we do put in a workaround then we'll probably need to restructure the overlap checks in both transferToTrustedChannel transferFromFileChannel to avoid overflow. Also transferFrom should check n >= 0 because 0 is a valid return. ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From bpb at openjdk.java.net Tue Aug 10 17:18:15 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 10 Aug 2021 17:18:15 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap [v2] In-Reply-To: References: Message-ID: > This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8140241: Fix potential overflow in interval overlap checks ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5061/files - new: https://git.openjdk.java.net/jdk/pull/5061/files/203495a5..f5f49956 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5061&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5061&range=00-01 Stats: 6 lines in 1 file changed: 1 ins; 0 del; 5 mod Patch: https://git.openjdk.java.net/jdk/pull/5061.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5061/head:pull/5061 PR: https://git.openjdk.java.net/jdk/pull/5061 From bpb at openjdk.java.net Tue Aug 10 17:18:15 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 10 Aug 2021 17:18:15 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap In-Reply-To: References: Message-ID: On Tue, 10 Aug 2021 01:21:21 GMT, Brian Burkhalter wrote: > This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. Fixed the overlap checks and the transferFrom n == 0 case. The scenario is unlikely but if it occurs the consequences are serious. I am trying to find out about the Apple bug (actually 2586576 as the one I filed, 37764443, was marked a duplicate). As an aside, I think that the cast to `int` at line 689 of `FileChannelImpl` should have been removed in the fix for [JDK-8271308](https://bugs.openjdk.java.net/browse/JDK-8271308) but was not. ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From alanb at openjdk.java.net Tue Aug 10 18:12:24 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Tue, 10 Aug 2021 18:12:24 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap In-Reply-To: References: Message-ID: On Tue, 10 Aug 2021 17:14:57 GMT, Brian Burkhalter wrote: > As an aside, I think that the cast to `int` at line 689 of `FileChannelImpl` should have been removed in the fix for [JDK-8271308](https://bugs.openjdk.java.net/browse/JDK-8271308) but was not. Including it as part of this change should be okay. ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From alanb at openjdk.java.net Tue Aug 10 18:15:29 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Tue, 10 Aug 2021 18:15:29 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap In-Reply-To: References: Message-ID: <1VRT--5J9bQ-_YPVq7HiSRtPXsakT_Tpfi65un0yqwg=.ea18532c-c93c-4ffd-8844-3810640bd862@github.com> On Tue, 10 Aug 2021 17:12:43 GMT, Brian Burkhalter wrote: > Fixed the overlap checks and the transferFrom n == 0 case. The overlap check looks right now although it is done without the positionLock so the position could potentially change (may not be worth worrying about). ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From bpb at openjdk.java.net Tue Aug 10 19:02:17 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 10 Aug 2021 19:02:17 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap [v3] In-Reply-To: References: Message-ID: > This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8140241: Use position lock in transferToTrustedChannel() overlap check ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5061/files - new: https://git.openjdk.java.net/jdk/pull/5061/files/f5f49956..5293edd2 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5061&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5061&range=01-02 Stats: 8 lines in 1 file changed: 2 ins; 0 del; 6 mod Patch: https://git.openjdk.java.net/jdk/pull/5061.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5061/head:pull/5061 PR: https://git.openjdk.java.net/jdk/pull/5061 From bpb at openjdk.java.net Tue Aug 10 19:02:20 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 10 Aug 2021 19:02:20 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap [v2] In-Reply-To: References: Message-ID: On Tue, 10 Aug 2021 17:18:15 GMT, Brian Burkhalter wrote: >> This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8140241: Fix potential overflow in interval overlap checks The issue is specific to macOS, but the test does not constrain the target platforms. Should it? ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From alanb at openjdk.java.net Wed Aug 11 06:11:22 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Wed, 11 Aug 2021 06:11:22 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap [v2] In-Reply-To: References: Message-ID: On Tue, 10 Aug 2021 18:59:20 GMT, Brian Burkhalter wrote: > The issue is specific to macOS, but the test does not constrain the target platforms. Should it? If it's reliable on all platforms then no reason to restrict it. I think we should at least using try-with-resources in main and generateBigFile so that it closes the file when there is an exception. ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From alanb at openjdk.java.net Wed Aug 11 11:30:28 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Wed, 11 Aug 2021 11:30:28 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: <5v2JxcXvfhlsA_lPDTS1u7LQul5S12tNrFHtN3lmnK8=.0186f02f-6537-4ae6-beda-a4268bc047a9@github.com> Message-ID: On Mon, 9 Aug 2021 18:10:52 GMT, Alan Bateman wrote: > I think I fixed all requested changes. Anymore comments on this PR? I've looked through the latest revision. Is here any way that we could drop most of the changes to ChannelInputStream and focus on one or two specific cases? I'm asking because there are several issues, inconsistencies, and it is trying to cover many scenarios that aren't covered by the test. If the original motivation was file -> file then it could be simplified down to a FileChannel -> FileChannel transfer as the default provider uses file channels. We could even push some support into FileChannelImpl so that it is done while holding the position lock. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From bpb at openjdk.java.net Wed Aug 11 15:49:55 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 11 Aug 2021 15:49:55 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap [v4] In-Reply-To: References: Message-ID: > This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8140241: Restructure test to use try-with-resources for file closing ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5061/files - new: https://git.openjdk.java.net/jdk/pull/5061/files/5293edd2..35386c14 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5061&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5061&range=02-03 Stats: 32 lines in 1 file changed: 7 ins; 5 del; 20 mod Patch: https://git.openjdk.java.net/jdk/pull/5061.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5061/head:pull/5061 PR: https://git.openjdk.java.net/jdk/pull/5061 From github.com+1701815+mkarg at openjdk.java.net Wed Aug 11 18:15:33 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Wed, 11 Aug 2021 18:15:33 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: <5v2JxcXvfhlsA_lPDTS1u7LQul5S12tNrFHtN3lmnK8=.0186f02f-6537-4ae6-beda-a4268bc047a9@github.com> Message-ID: On Wed, 11 Aug 2021 11:27:53 GMT, Alan Bateman wrote: > I've looked through the latest revision. Is there any way that we could drop most of the changes to ChannelInputStream and focus on one or two specific cases? I'm asking because there are several issues, inconsistencies, and it is trying to cover many scenarios that aren't covered by the test. > If the original motivation was file -> file then it could be simplified down to a FileChannel -> FileChannel transfer as the default provider uses file channels. We could even push some support into FileChannelImpl so that it is done while holding the position lock. I am a bit disappointed actually about that destructive answer at that late point in time, now that I worked for months on all the requested changes and tests. To prevent exactly this situation, I deliberately had the discussion started in JIRA only, and I deliberately had the original code being just a draft in the first place, and I deliberately did nearly *everything* I was asked to (including even the most irrelevant minor code style issues). And you come up with the request to drop the code **now**? Certainly we could reduce the PR to just file channels, but in fact, now that I spent all the time in the non-file-channels, I wonder why I shall throw away all that work and go just with file channels actually? What is not covered that was originally covered, and what is that lots of issues you talk about? Actually I cannot see the actual problem unless you name it. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From alanb at openjdk.java.net Wed Aug 11 18:57:24 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Wed, 11 Aug 2021 18:57:24 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap [v4] In-Reply-To: References: Message-ID: <7yq-_eZ5GZggedtp3732roSz14qAolQAaikS9JSNXfY=.b1cc6074-6b28-43ad-b6de-4bb6c708f237@github.com> On Wed, 11 Aug 2021 15:49:55 GMT, Brian Burkhalter wrote: >> This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8140241: Restructure test to use try-with-resources for file closing Marked as reviewed by alanb (Reviewer). src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 591: > 589: synchronized (positionLock) { > 590: long posThis = position(); > 591: if (posThis - count + 1 <= position && I realise I suggested we need the positionLock here but I don't think it is needed now because there is only one call to position(). src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 803: > 801: > 802: if (src instanceof FileChannelImpl) { > 803: long n = transferFromFileChannel((FileChannelImpl)src, position, count); It might be a bit cleaner to use the pattern matching with the instanceof to avoid the cast to FileChannelImpl. ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From bpb at openjdk.java.net Wed Aug 11 19:26:28 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 11 Aug 2021 19:26:28 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap [v4] In-Reply-To: <7yq-_eZ5GZggedtp3732roSz14qAolQAaikS9JSNXfY=.b1cc6074-6b28-43ad-b6de-4bb6c708f237@github.com> References: <7yq-_eZ5GZggedtp3732roSz14qAolQAaikS9JSNXfY=.b1cc6074-6b28-43ad-b6de-4bb6c708f237@github.com> Message-ID: On Wed, 11 Aug 2021 18:53:53 GMT, Alan Bateman wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8140241: Restructure test to use try-with-resources for file closing > > src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 591: > >> 589: synchronized (positionLock) { >> 590: long posThis = position(); >> 591: if (posThis - count + 1 <= position && > > I realise I suggested we need the positionLock here but I don't think it is needed now because there is only one call to position(). Will remove. > src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 803: > >> 801: >> 802: if (src instanceof FileChannelImpl) { >> 803: long n = transferFromFileChannel((FileChannelImpl)src, position, count); > > It might be a bit cleaner to use the pattern matching with the instanceof to avoid the cast to FileChannelImpl. Right, I should have caught that myself. ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From bpb at openjdk.java.net Wed Aug 11 19:50:49 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 11 Aug 2021 19:50:49 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap [v5] In-Reply-To: References: Message-ID: > This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8140241: Use pattern matching; remove sync on position lock ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5061/files - new: https://git.openjdk.java.net/jdk/pull/5061/files/35386c14..bf431122 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5061&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5061&range=03-04 Stats: 9 lines in 1 file changed: 0 ins; 2 del; 7 mod Patch: https://git.openjdk.java.net/jdk/pull/5061.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5061/head:pull/5061 PR: https://git.openjdk.java.net/jdk/pull/5061 From bpb at openjdk.java.net Thu Aug 12 01:13:32 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 12 Aug 2021 01:13:32 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: Message-ID: On Sun, 1 Aug 2021 22:01:33 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has updated the pull request incrementally with two additional commits since the last revision: > > - Draft: Eliminated duplicate code using lambda expressions > - Draft: Use blocking mode also for target channel src/java.base/share/classes/sun/nio/ch/ChannelInputStream.java line 175: > 173: throw new IllegalBlockingModeException(); > 174: } > 175: return tls.supply(); It does not look like the `SelectableChannel` branch is tested, including whether an `IllegalBlockingModeException` is thrown when it should be. src/java.base/share/classes/sun/nio/ch/ChannelInputStream.java line 249: > 247: } > 248: > 249: private static long transfer(ReadableByteChannel src, WritableByteChannel dst) throws IOException { Does this method have a measurable improvement in performance over `InputStream.transferTo()`? test/jdk/sun/nio/ch/ChannelInputStream/TransferTo.java line 93: > 91: checkTransferredContents(inputStreamProvider, outputStreamProvider, createRandomBytes(1024, 4096)); > 92: // to span through several batches > 93: checkTransferredContents(inputStreamProvider, outputStreamProvider, createRandomBytes(16384, 16384)); Should some random-sized buffers be tested? (Use `jdk.test.lib.RandomFactory` factory for this, not `j.u.Random`. The existing use of `Random` is fine.) Should some testing be done of streams with non-zero initial position? test/jdk/sun/nio/ch/ChannelInputStream/TransferTo.java line 101: > 99: try (InputStream in = inputStreamProvider.input(inBytes); > 100: OutputStream out = outputStreamProvider.output(recorder::set)) { > 101: in.transferTo(out); The return value of `transferTo()` is not checked. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From alanb at openjdk.java.net Thu Aug 12 07:49:22 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Thu, 12 Aug 2021 07:49:22 GMT Subject: RFR: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap [v5] In-Reply-To: References: Message-ID: On Wed, 11 Aug 2021 19:50:49 GMT, Brian Burkhalter wrote: >> This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8140241: Use pattern matching; remove sync on position lock Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From alanb at openjdk.java.net Thu Aug 12 08:43:34 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Thu, 12 Aug 2021 08:43:34 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: Message-ID: <9K3gZpDYVCFjt7WjCfW6YJKmULOHw7cv-doIFWeoI-E=.b4a84b0d-a448-4d0a-a54a-0756c899fd69@github.com> On Sun, 1 Aug 2021 22:01:33 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has updated the pull request incrementally with two additional commits since the last revision: > > - Draft: Eliminated duplicate code using lambda expressions > - Draft: Use blocking mode also for target channel > I am a bit disappointed actually about that destructive answer at that late point in time, now that I worked for months on all the requested changes and tests. To prevent exactly this situation, I deliberately had the discussion started in JIRA only, and I deliberately had the original code being just a draft in the first place, and I deliberately did nearly _everything_ I was asked to (including even the most irrelevant minor code style issues). And you come up with the request to drop the code **now**? > > Certainly we could reduce the PR to just file channels, but in fact, now that I spent all the time in the non-file-channels, I wonder why I shall throw away all that work and go just with file channels actually? What is not covered that was originally covered, and what is that lots of issues you talk about? Actually I cannot see the actual problem unless you name it. There are 78 comments on this PR so far. We've tried to point out the bugs and issues at each iteration. We asked for tests because the changes introduce several code paths and implementations that would not be exercised by existing tests. There are several scenarios still missing and the patch doesn't yet have the microbenchmarks to demonstrate the improvements. I assume this is your first contribution so there will be learning curve and maybe some frustration. I think you have a better chance of success if you split this up and reduce the scope of this PR down to something manageable. Keeping the selectable channels out of this PR and focusing on the case where the input and output streams wrap file channels should make it simpler and may lead to a better solution. Reducing the scope will also reduce the burden on reviewers. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From bpb at openjdk.java.net Thu Aug 12 08:43:33 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 12 Aug 2021 08:43:33 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: <5v2JxcXvfhlsA_lPDTS1u7LQul5S12tNrFHtN3lmnK8=.0186f02f-6537-4ae6-beda-a4268bc047a9@github.com> Message-ID: On Wed, 11 Aug 2021 11:27:53 GMT, Alan Bateman wrote: >>> I think I fixed all requested changes. Anymore comments on this PR? >> >> I hope to get to this soon. > >> I think I fixed all requested changes. Anymore comments on this PR? > > I've looked through the latest revision. Is there any way that we could drop most of the changes to ChannelInputStream and focus on one or two specific cases? I'm asking because there are several issues, inconsistencies, and it is trying to cover many scenarios that aren't covered by the test. > > If the original motivation was file -> file then it could be simplified down to a FileChannel -> FileChannel transfer as the default provider uses file channels. We could even push some support into FileChannelImpl so that it is done while holding the position lock. I do not know exactly what @AlanBateman had in mind, but I think there is general concern about ensuring that all combinations of channel types and all execution paths are exercised. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From bpb at openjdk.java.net Thu Aug 12 15:30:28 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 12 Aug 2021 15:30:28 GMT Subject: Integrated: 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap In-Reply-To: References: Message-ID: On Tue, 10 Aug 2021 01:21:21 GMT, Brian Burkhalter wrote: > This request proposes to fix a failure on macOS which occurs when transferring to or from a file channel from or to the same file channel when the source and destination intervals overlap. This pull request has now been integrated. Changeset: 428d5169 Author: Brian Burkhalter URL: https://git.openjdk.java.net/jdk/commit/428d51694f56788f89e8df100a74cbadd369ffa6 Stats: 167 lines in 6 files changed: 160 ins; 0 del; 7 mod 8140241: (fc) Data transfer from FileChannel to itself causes hang in case of overlap Reviewed-by: alanb ------------- PR: https://git.openjdk.java.net/jdk/pull/5061 From lancea at openjdk.java.net Thu Aug 12 17:50:41 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Thu, 12 Aug 2021 17:50:41 GMT Subject: RFR: 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path Message-ID: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> Hi all, Please review the fix for JDK-8263940 to address an issues when the default file system provider is packaged as JAR file on class path. The patch also addresses the `@bug` line for JDK-8271194 Mach5 Tier1 - Tier3 have run without issues Best, Lance ------------- Commit messages: - Fix for JDK-8263940 Changes: https://git.openjdk.java.net/jdk/pull/5103/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5103&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8263940 Stats: 55 lines in 2 files changed: 49 ins; 2 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/5103.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5103/head:pull/5103 PR: https://git.openjdk.java.net/jdk/pull/5103 From github.com+1701815+mkarg at openjdk.java.net Thu Aug 12 18:15:32 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 12 Aug 2021 18:15:32 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: <9K3gZpDYVCFjt7WjCfW6YJKmULOHw7cv-doIFWeoI-E=.b4a84b0d-a448-4d0a-a54a-0756c899fd69@github.com> References: <9K3gZpDYVCFjt7WjCfW6YJKmULOHw7cv-doIFWeoI-E=.b4a84b0d-a448-4d0a-a54a-0756c899fd69@github.com> Message-ID: On Thu, 12 Aug 2021 08:40:34 GMT, Alan Bateman wrote: >> Markus KARG has updated the pull request incrementally with two additional commits since the last revision: >> >> - Draft: Eliminated duplicate code using lambda expressions >> - Draft: Use blocking mode also for target channel > >> I am a bit disappointed actually about that destructive answer at that late point in time, now that I worked for months on all the requested changes and tests. To prevent exactly this situation, I deliberately had the discussion started in JIRA only, and I deliberately had the original code being just a draft in the first place, and I deliberately did nearly _everything_ I was asked to (including even the most irrelevant minor code style issues). And you come up with the request to drop the code **now**? >> >> Certainly we could reduce the PR to just file channels, but in fact, now that I spent all the time in the non-file-channels, I wonder why I shall throw away all that work and go just with file channels actually? What is not covered that was originally covered, and what is that lots of issues you talk about? Actually I cannot see the actual problem unless you name it. > > There are 78 comments on this PR so far. We've tried to point out the bugs and issues at each iteration. We asked for tests because the changes introduce several code paths and implementations that would not be exercised by existing tests. There are several scenarios still missing and the patch doesn't yet have the microbenchmarks to demonstrate the improvements. > > I assume this is your first contribution so there will be learning curve and maybe some frustration. I think you have a better chance of success if you split this up and reduce the scope of this PR down to something manageable. Keeping the selectable channels out of this PR and focusing on the case where the input and output streams wrap file channels should make it simpler and may lead to a better solution. Reducing the scope will also reduce the burden on reviewers. > I do not know exactly what @AlanBateman had in mind, but I think there is general concern about ensuring that all combinations of channel types and all execution paths are exercised. @AlanBateman @bplb Does it make **any** real sense to answer your recent questions, provide the proofs, tests and benchmark results (I actually would love to *if* it makes sense) *or* will the outcome be that I *must* drop everything besides file channels *anyways* (In that case it is in vain)? As my time is just as precious as yours I really need to know that **before** I spend more weeks into code paths that you possibly decided to never accept ever. Don't get me wrong, if you see a chance to keep the code once I provided the answers I will do that, but if you do not see that chance, please frankly and unambiguously tell me **now**. Thanks. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From naoto at openjdk.java.net Thu Aug 12 18:59:25 2021 From: naoto at openjdk.java.net (Naoto Sato) Date: Thu, 12 Aug 2021 18:59:25 GMT Subject: RFR: 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path In-Reply-To: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> References: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> Message-ID: On Thu, 12 Aug 2021 17:43:48 GMT, Lance Andersen wrote: > Hi all, > > Please review the fix for JDK-8263940 to address an issues when the default file system provider is packaged as JAR file on class path. > > The patch also addresses the `@bug` line for JDK-8271194 > > Mach5 Tier1 - Tier3 have run without issues > > Best, > Lance Looks good to me, Lance. test/jdk/java/nio/file/spi/SetDefaultProvider.java line 107: > 105: .map(path -> path.getFileName().toString()) > 106: .filter(f -> f.startsWith("TestProvider")) > 107: .collect(Collectors.toList()); Nit: Could simply issue `.toList()` here. ------------- Marked as reviewed by naoto (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/5103 From bpb at openjdk.java.net Thu Aug 12 19:27:43 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 12 Aug 2021 19:27:43 GMT Subject: RFR: 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path [v2] In-Reply-To: References: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> Message-ID: On Thu, 12 Aug 2021 19:24:39 GMT, Lance Andersen wrote: >> Hi all, >> >> Please review the fix for JDK-8263940 to address an issues when the default file system provider is packaged as JAR file on class path. >> >> The patch also addresses the `@bug` line for JDK-8271194 >> >> Mach5 Tier1 - Tier3 have run without issues >> >> Best, >> Lance > > Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: > > Use toList() Marked as reviewed by bpb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/5103 From lancea at openjdk.java.net Thu Aug 12 19:27:42 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Thu, 12 Aug 2021 19:27:42 GMT Subject: RFR: 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path [v2] In-Reply-To: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> References: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> Message-ID: > Hi all, > > Please review the fix for JDK-8263940 to address an issues when the default file system provider is packaged as JAR file on class path. > > The patch also addresses the `@bug` line for JDK-8271194 > > Mach5 Tier1 - Tier3 have run without issues > > Best, > Lance Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: Use toList() ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5103/files - new: https://git.openjdk.java.net/jdk/pull/5103/files/c09d7c32..6a65fb35 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5103&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5103&range=00-01 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/5103.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5103/head:pull/5103 PR: https://git.openjdk.java.net/jdk/pull/5103 From lancea at openjdk.java.net Thu Aug 12 19:27:44 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Thu, 12 Aug 2021 19:27:44 GMT Subject: RFR: 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path [v2] In-Reply-To: References: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> Message-ID: On Thu, 12 Aug 2021 18:55:08 GMT, Naoto Sato wrote: >> Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: >> >> Use toList() > > test/jdk/java/nio/file/spi/SetDefaultProvider.java line 107: > >> 105: .map(path -> path.getFileName().toString()) >> 106: .filter(f -> f.startsWith("TestProvider")) >> 107: .collect(Collectors.toList()); > > Nit: Could simply issue `.toList()` here. Thank you Naoto, it is a bit cleaner so pushed the update. ------------- PR: https://git.openjdk.java.net/jdk/pull/5103 From iris at openjdk.java.net Thu Aug 12 21:42:25 2021 From: iris at openjdk.java.net (Iris Clark) Date: Thu, 12 Aug 2021 21:42:25 GMT Subject: RFR: 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path [v2] In-Reply-To: References: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> Message-ID: On Thu, 12 Aug 2021 19:27:42 GMT, Lance Andersen wrote: >> Hi all, >> >> Please review the fix for JDK-8263940 to address an issues when the default file system provider is packaged as JAR file on class path. >> >> The patch also addresses the `@bug` line for JDK-8271194 >> >> Mach5 Tier1 - Tier3 have run without issues >> >> Best, >> Lance > > Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: > > Use toList() Marked as reviewed by iris (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/5103 From joehw at openjdk.java.net Thu Aug 12 22:24:24 2021 From: joehw at openjdk.java.net (Joe Wang) Date: Thu, 12 Aug 2021 22:24:24 GMT Subject: RFR: 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path [v2] In-Reply-To: References: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> Message-ID: On Thu, 12 Aug 2021 19:27:42 GMT, Lance Andersen wrote: >> Hi all, >> >> Please review the fix for JDK-8263940 to address an issues when the default file system provider is packaged as JAR file on class path. >> >> The patch also addresses the `@bug` line for JDK-8271194 >> >> Mach5 Tier1 - Tier3 have run without issues >> >> Best, >> Lance > > Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: > > Use toList() Marked as reviewed by joehw (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/5103 From alanb at openjdk.java.net Fri Aug 13 12:57:29 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 13 Aug 2021 12:57:29 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: Message-ID: On Sun, 1 Aug 2021 22:01:33 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has updated the pull request incrementally with two additional commits since the last revision: > > - Draft: Eliminated duplicate code using lambda expressions > - Draft: Use blocking mode also for target channel > Does it make **any** real sense to answer your recent questions, provide the proofs, tests and benchmark results (I actually would love to _if_ it makes sense) _or_ will the outcome be that I _must_ drop everything besides file channels _anyways_ (In that case it is in vain)? As my time is just as precious as yours I really need to know that **before** I spend more weeks into code paths that you possibly decided to never accept ever. Don't get me wrong, if you see a chance to keep the code once I provided the answers I will do that, but if you do not see that chance, please frankly and unambiguously tell me **now**. Thanks. I think the best course of action is to reduce the scope of this PR to the file channel cases. There is no reason why future PRs can't build on this and add implementations for other channel types. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From lancea at openjdk.java.net Fri Aug 13 16:14:24 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Fri, 13 Aug 2021 16:14:24 GMT Subject: Integrated: 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path In-Reply-To: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> References: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> Message-ID: On Thu, 12 Aug 2021 17:43:48 GMT, Lance Andersen wrote: > Hi all, > > Please review the fix for JDK-8263940 to address an issues when the default file system provider is packaged as JAR file on class path. > > The patch also addresses the `@bug` line for JDK-8271194 > > Mach5 Tier1 - Tier3 have run without issues > > Best, > Lance This pull request has now been integrated. Changeset: 717792c3 Author: Lance Andersen URL: https://git.openjdk.java.net/jdk/commit/717792c3b728584413572e7aede83290779be2a2 Stats: 55 lines in 2 files changed: 49 ins; 2 del; 4 mod 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path Reviewed-by: naoto, bpb, iris, joehw ------------- PR: https://git.openjdk.java.net/jdk/pull/5103 From alanb at openjdk.java.net Fri Aug 13 17:46:26 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 13 Aug 2021 17:46:26 GMT Subject: RFR: 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path [v2] In-Reply-To: References: <3S_DNN9eJHOO64aa2RmCliXhv9P6eyk-Y32BhAACbjs=.e5267400-28e2-4ee7-8237-25321fb88e16@github.com> Message-ID: On Thu, 12 Aug 2021 19:27:42 GMT, Lance Andersen wrote: >> Hi all, >> >> Please review the fix for JDK-8263940 to address an issues when the default file system provider is packaged as JAR file on class path. >> >> The patch also addresses the `@bug` line for JDK-8271194 >> >> Mach5 Tier1 - Tier3 have run without issues >> >> Best, >> Lance > > Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: > > Use toList() test/jdk/java/nio/file/spi/SetDefaultProvider.java line 26: > 24: /** > 25: * @test > 26: * @bug 4313887 7006126 8142968 8178380 8183320 8210112 8266345 8263940 Thanks for correcting the @bug tag. test/jdk/java/nio/file/spi/SetDefaultProvider.java line 89: > 87: createFileSystemProviderJar(jar, Path.of(testClasses)); > 88: String classpath = jar + File.pathSeparator + testClasses > 89: + File.separator + "modules" + File.separator + "m"; This ends up with two copies of TestFIleSystemProvider on the class path. I think we should compile TestProvider to a different directory. That will eliminate the need to filter the classes when creating the JAR file. test/jdk/java/nio/file/spi/SetDefaultProvider.java line 99: > 97: */ > 98: private void createFileSystemProviderJar(Path jar, Path dir) throws IOException { > 99: In this test, the supporting methods are at the end of the source file, probably should keep it consistent. ------------- PR: https://git.openjdk.java.net/jdk/pull/5103 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 15 13:40:36 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 15 Aug 2021 13:40:36 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: Message-ID: On Fri, 13 Aug 2021 12:54:55 GMT, Alan Bateman wrote: > > Does it make **any** real sense to answer your recent questions, provide the proofs, tests and benchmark results (I actually would love to _if_ it makes sense) _or_ will the outcome be that I _must_ drop everything besides file channels _anyways_ (In that case it is in vain)? As my time is just as precious as yours I really need to know that **before** I spend more weeks into code paths that you possibly decided to never accept ever. Don't get me wrong, if you see a chance to keep the code once I provided the answers I will do that, but if you do not see that chance, please frankly and unambiguously tell me **now**. Thanks. > > I think the best course of action is to reduce the scope of this PR to the file channel cases. There is no reason why future PRs can't build on this and add implementations for other channel types. Agreed, I will split this PR into several PRs, so we can start with the low-hanging fruits first and later dive into the more complex cases. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From bpb at openjdk.java.net Wed Aug 18 02:39:34 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 18 Aug 2021 02:39:34 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted Message-ID: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. ------------- Commit messages: - 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted Changes: https://git.openjdk.java.net/jdk/pull/5154/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5154&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8265261 Stats: 26 lines in 1 file changed: 18 ins; 0 del; 8 mod Patch: https://git.openjdk.java.net/jdk/pull/5154.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5154/head:pull/5154 PR: https://git.openjdk.java.net/jdk/pull/5154 From dfuchs at openjdk.java.net Wed Aug 18 09:11:24 2021 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Wed, 18 Aug 2021 09:11:24 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: <_3Xi4_l9fyq04AtcdLqCQ1BRQd5j69aMV48qdqnFsAI=.ddccdac9-5837-4bf0-bbd7-7f6e45854e69@github.com> On Wed, 18 Aug 2021 02:32:53 GMT, Brian Burkhalter wrote: > This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. LGTM ------------- Marked as reviewed by dfuchs (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/5154 From msheppar at openjdk.java.net Wed Aug 18 10:06:22 2021 From: msheppar at openjdk.java.net (Mark Sheppard) Date: Wed, 18 Aug 2021 10:06:22 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Wed, 18 Aug 2021 02:32:53 GMT, Brian Burkhalter wrote: > This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. is a countdown of 2 required? or is it sufficient that the Interrupter thread should wait for the main to tell it to proceed? Does the fact that the test runs agentvm influence the scheduling of the test's threads or is is that irrelevant on linux? ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Wed Aug 18 15:58:22 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 18 Aug 2021 15:58:22 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Wed, 18 Aug 2021 02:32:53 GMT, Brian Burkhalter wrote: > This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. It did indeed look as if the copy starts before the interrupter thread but that is not guaranteed. I do not know about the difference between agentvm and othervm. Another alternative would be to repeatedly interrupt at a fixed rate with zero delay but I don't know whether the original intent was to have only one interrupt. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From msheppar at openjdk.java.net Wed Aug 18 21:31:23 2021 From: msheppar at openjdk.java.net (Mark Sheppard) Date: Wed, 18 Aug 2021 21:31:23 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: <_La76uJAH7QpH4Nm0xvMetuWlS6SkAxzNL_DwwmXxkY=.5bc15800-9030-4304-a7b8-865ce4be8821@github.com> On Wed, 18 Aug 2021 02:32:53 GMT, Brian Burkhalter wrote: > This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. ah yes! with the count of 2 you wish to avoid either thread racing ahead, and so are attempting to arrange that the interrupter is at least Ready to Run when the copy thread initiates its copy, as such creating a reasonable probability that the interrupter will invoke the interrupt while the copy is in progress. As such, the ping pong between the two threads is a sort of ad hoc barrier, which seems reasonable and will most likely work the majority of the time. But there is, I think, still some probability of missing the interrupt, if the interrupter thread "got stuck" after its little sleep, and was less than fairly scheduled? This is where agentvm or othervm may have influence, depending on the scheduling algorithm of the OS. Are threads, on Windows, scheduled independent of their owning process or relative to their owning process. othervm mode, afaik, launches a new process to executed the test, not sure if that would influence the scheduling of its threads. Would some additional diagnostic output be useful in the Interrupter thread to indicate that it starting to run and it is invoking interrupt or has invoked interrupt? In the second phase of the test the cancellation of scheduled task, is a latch also required in this instance, to ensure that when the cancel is invoked that the copy is in progress. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Wed Aug 18 21:44:21 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 18 Aug 2021 21:44:21 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: <_La76uJAH7QpH4Nm0xvMetuWlS6SkAxzNL_DwwmXxkY=.5bc15800-9030-4304-a7b8-865ce4be8821@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> <_La76uJAH7QpH4Nm0xvMetuWlS6SkAxzNL_DwwmXxkY=.5bc15800-9030-4304-a7b8-865ce4be8821@github.com> Message-ID: On Wed, 18 Aug 2021 21:28:42 GMT, Mark Sheppard wrote: >> This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. > > ah yes! with the count of 2 you wish to avoid either thread racing ahead, and so are attempting to arrange that the interrupter is at least Ready to Run when the copy thread initiates its copy, as such creating a reasonable probability that the interrupter will invoke the interrupt while the copy is in progress. As such, the ping pong between the two threads is a sort of ad hoc barrier, which seems reasonable and will most likely work the majority of the time. > > But there is, I think, still some probability of missing the interrupt, if the interrupter thread "got stuck" after its little sleep, and was less than fairly scheduled? This is where agentvm or othervm may have influence, depending on the scheduling algorithm of the OS. Are threads, on Windows, scheduled independent of their owning process or relative to their owning process. othervm mode, afaik, launches a new process to executed the test, not sure if that would influence the scheduling of its threads. > > Would some additional diagnostic output be useful in the Interrupter thread to indicate that it starting to run and it is invoking interrupt or has invoked interrupt? > > In the second phase of the test the cancellation of scheduled task, is a latch also required in this instance, to ensure that when the cancel is invoked that the copy is in progress. @msheppar Right, the latch is intended to do exactly what you described. There is definitely still some probability of missing the interrupt however, but this results in a failure only if the copy took more than `DURATION_MAX_IN_MS` (5 seconds). I don't know how this limit was determined. I think you are correct about the problem of the interrupter getting stuck. I don't know which vm mode would best minimize this problem. I had some print statements in the test when I was playing around with different approaches, but I was afraid that they might affect the timing. I'll take another look at it however as well as the idea of a latch added to the cancellation section. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Wed Aug 18 23:34:24 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 18 Aug 2021 23:34:24 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Wed, 18 Aug 2021 02:32:53 GMT, Brian Burkhalter wrote: > This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. Strangely enough it looks like no exception is thrown if cancellation fails. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Thu Aug 19 01:06:47 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 19 Aug 2021 01:06:47 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v2] In-Reply-To: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: > This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8265261: Fix the section which tests cancellation ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5154/files - new: https://git.openjdk.java.net/jdk/pull/5154/files/7c75978b..85f78867 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5154&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5154&range=00-01 Stats: 19 lines in 1 file changed: 11 ins; 0 del; 8 mod Patch: https://git.openjdk.java.net/jdk/pull/5154.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5154/head:pull/5154 PR: https://git.openjdk.java.net/jdk/pull/5154 From github.com+1701815+mkarg at openjdk.java.net Thu Aug 19 06:23:44 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 19 Aug 2021 06:23:44 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() Message-ID: As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). ------------- Commit messages: - 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() Changes: https://git.openjdk.java.net/jdk/pull/5179/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8265891 Stats: 449 lines in 4 files changed: 369 ins; 76 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/5179.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5179/head:pull/5179 PR: https://git.openjdk.java.net/jdk/pull/5179 From github.com+1701815+mkarg at openjdk.java.net Thu Aug 19 06:23:45 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 19 Aug 2021 06:23:45 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() In-Reply-To: References: Message-ID: On Thu, 19 Aug 2021 06:16:42 GMT, Markus KARG wrote: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). @AlanBateman @bplb This PR is a spin-off for FileChannels only, as suggested by you. I kindly request your review. Thanks. ------------- PR: https://git.openjdk.java.net/jdk/pull/5179 From github.com+1701815+mkarg at openjdk.java.net Thu Aug 19 06:24:29 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 19 Aug 2021 06:24:29 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: Message-ID: On Sun, 15 Aug 2021 13:37:23 GMT, Markus KARG wrote: > I think the best course of action is to reduce the scope of this PR to the file channel cases. There is no reason why future PRs can't build on this and add implementations for other channel types. I have split up this PR so that only the lowest hanging fruit is covered and kindly request review: https://github.com/openjdk/jdk/pull/5179. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From dfuchs at openjdk.java.net Thu Aug 19 11:07:22 2021 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Thu, 19 Aug 2021 11:07:22 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v2] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Thu, 19 Aug 2021 01:06:47 GMT, Brian Burkhalter wrote: >> This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8265261: Fix the section which tests cancellation Updated logic LGTM. test/jdk/java/nio/file/Files/InterruptCopy.java line 144: > 142: else { > 143: result.get(); > 144: throw new RuntimeException("Copy was not cancelled"); Should `DURATION_MAX_IN_MS` be taken into account here too to decide whether to throw or not? ------------- Marked as reviewed by dfuchs (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/5154 From alanb at openjdk.java.net Thu Aug 19 13:13:25 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Thu, 19 Aug 2021 13:13:25 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() In-Reply-To: References: Message-ID: <4aYyUY5raNV7PBhtlbPL9Mrqaoy4eeLct4uoq4jI1KI=.500ab3f2-1981-4a4d-8ea5-52da621f3a4d@github.com> On Thu, 19 Aug 2021 06:16:42 GMT, Markus KARG wrote: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Thanks for the update. I see you've kept the code for the file -> socket scenario but there aren't any tests for this. If you do want to include this scenario then we'll need at least a test for the case where the output stream wraps a socket channel that is configured blocking. The test/jdk/java/nio/channels/Channels directory would be a good location for the new test because it is testing the input stream returned by the Channels API. As regards ChannelInputStream then I had hoped that ThrowingLongSupplier would go away. If you keep your 2-arg transfer method then we should be able to implementation transferTo very simply, something like public long transferTo(OutputStream out) throws IOException { if (ch instanceof FileChannel fc && out instanceof ChannelOutputStream cos) { WritableByteChannel target = cos.channel(); if (target instanceof SelectableChannel sc) { synchronized (sc.blockingLock()) { if (!sc.isBlocking()) throw new IllegalBlockingModeException(); return transfer(fc, target); } } return transfer(fc, target); } return super.transferTo(out); } ------------- PR: https://git.openjdk.java.net/jdk/pull/5179 From bpb at openjdk.java.net Thu Aug 19 15:35:27 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 19 Aug 2021 15:35:27 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v2] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Thu, 19 Aug 2021 11:04:05 GMT, Daniel Fuchs wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8265261: Fix the section which tests cancellation > > test/jdk/java/nio/file/Files/InterruptCopy.java line 144: > >> 142: else { >> 143: result.get(); >> 144: throw new RuntimeException("Copy was not cancelled"); > > Should `DURATION_MAX_IN_MS` be taken into account here too to decide whether to throw or not? I don't know. It seems like cancellation ought to be a bit more predictable than interruption. I am not sure how `DURATION_MAX_IN_MS` was determined. If there were no "hiccups" in the threads starting, then with the correct timing I am not sure that such a threshold is even needed. The execution time for the copy that I have observed is between 150ms (macOS) and 400ms (Windows) which is a lot less than 5s, so there would have to be some slowdown for this threshold even to be reached. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From github.com+1701815+mkarg at openjdk.java.net Thu Aug 19 17:20:24 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 19 Aug 2021 17:20:24 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() In-Reply-To: <4aYyUY5raNV7PBhtlbPL9Mrqaoy4eeLct4uoq4jI1KI=.500ab3f2-1981-4a4d-8ea5-52da621f3a4d@github.com> References: <4aYyUY5raNV7PBhtlbPL9Mrqaoy4eeLct4uoq4jI1KI=.500ab3f2-1981-4a4d-8ea5-52da621f3a4d@github.com> Message-ID: On Thu, 19 Aug 2021 13:10:25 GMT, Alan Bateman wrote: > Thanks for the update. I see you've kept the code for the file -> socket scenario but there aren't any tests for this. Seems I misunderstood your last week's request of separating the PRs! I simply split up the features into several PRs. So you *actually* wanted to start with **just** the FC->FC use case only (which, actually, was not an explicit execution path of the original PR)? I will further narrow this PR then, no problem, just tell me! And do I understand correctly, you do not just want to have *some* test for each PR that just proofs **the API** is correctly implemented (= Blackbox), but you actually want to have one test for *each existing implementation* of `WriteableByteChannel` (= Whitebox)? I mean, as it is an interface which can be implemented by *infinite* classes by *anybody*, such a request is inherently unresolvable. Or do you just want to have tests for *specific* implementations, then, besides `SocketChannel`, which ones do you want to have tested? ------------- PR: https://git.openjdk.java.net/jdk/pull/5179 From github.com+1701815+mkarg at openjdk.java.net Thu Aug 19 17:50:28 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 19 Aug 2021 17:50:28 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() In-Reply-To: <4aYyUY5raNV7PBhtlbPL9Mrqaoy4eeLct4uoq4jI1KI=.500ab3f2-1981-4a4d-8ea5-52da621f3a4d@github.com> References: <4aYyUY5raNV7PBhtlbPL9Mrqaoy4eeLct4uoq4jI1KI=.500ab3f2-1981-4a4d-8ea5-52da621f3a4d@github.com> Message-ID: On Thu, 19 Aug 2021 13:10:25 GMT, Alan Bateman wrote: > I had hoped that ThrowingLongSupplier would go away. If you keep your 2-arg transfer method then we should be able to implementation transferTo very simply, something like > > ``` > public long transferTo(OutputStream out) throws IOException { > if (ch instanceof FileChannel fc > && out instanceof ChannelOutputStream cos) { > WritableByteChannel target = cos.channel(); > if (target instanceof SelectableChannel sc) { > synchronized (sc.blockingLock()) { > if (!sc.isBlocking()) > throw new IllegalBlockingModeException(); > return transfer(fc, target); > } > } > return transfer(fc, target); > } > return super.transferTo(out); > } > ``` This is not true. It will only check the blocking mode for the target. But technically it is possible that someone writes an implementation of FileChannel which implements SelectableChannel. In that case it is needed to check the blocking mode on the source, too. So either we (a) ignore this possibility, (b) write a very complex code which calls transfer(fc, target) in FOUR code paths not just two, or (c) we keep the ThrowingLongSupplier as an elegant way to perform the 2x2 matrix. ------------- PR: https://git.openjdk.java.net/jdk/pull/5179 From msheppar at openjdk.java.net Thu Aug 19 17:54:23 2021 From: msheppar at openjdk.java.net (Mark Sheppard) Date: Thu, 19 Aug 2021 17:54:23 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v2] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Thu, 19 Aug 2021 15:32:15 GMT, Brian Burkhalter wrote: >> test/jdk/java/nio/file/Files/InterruptCopy.java line 144: >> >>> 142: else { >>> 143: result.get(); >>> 144: throw new RuntimeException("Copy was not cancelled"); >> >> Should `DURATION_MAX_IN_MS` be taken into account here too to decide whether to throw or not? > > I don't know. It seems like cancellation ought to be a bit more predictable than interruption. I am not sure how `DURATION_MAX_IN_MS` was determined. If there were no "hiccups" in the threads starting, then with the correct timing I am not sure that such a threshold is even needed. The execution time for the copy that I have observed is between 150ms (macOS) and 400ms (Windows) which is a lot less than 5s, so there would have to be some slowdown for this threshold even to be reached. Not wishing to labour the discussion on the change, which will in the main deal with the issue, but is it not the case with the current set of Intermittent failures that there is this slowdown, i.e. the copy is taking longer than the threshold of 5 seconds hence the RuntimeException with Copy was not Interrupted? As such, if (duration > DURATION_MAX_IN_MS) throw new RuntimeException("Copy was not interrupted"); is triggering a failure. You would expect if a typical copy duration is 400msecs that if the duration is taking a lot longer that the interrupt has a greater probability to happen. the rationale for this logic is not clear - so if the duration is less than the copy threshold then test proceeds as if an interrupt has occurred? the success and failure semantics for the test are not absolute. As such, maybe there's an opportunity to be more exact with test semantics e.g. retrying to interrupt the copy if it didn't happen. as an aside it would appear that invoking Thread.interrupted(); in the main thread seems redundant ? ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From alanb at openjdk.java.net Thu Aug 19 18:28:29 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Thu, 19 Aug 2021 18:28:29 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() In-Reply-To: References: <4aYyUY5raNV7PBhtlbPL9Mrqaoy4eeLct4uoq4jI1KI=.500ab3f2-1981-4a4d-8ea5-52da621f3a4d@github.com> Message-ID: On Thu, 19 Aug 2021 17:47:00 GMT, Markus KARG wrote: > This is not true. It will only check the blocking mode for the target. But technically it is possible that someone writes an implementation of FileChannel which implements SelectableChannel. In that case it is needed to check the blocking mode on the source, too. So either we (a) ignore this possibility, (b) write a very complex code which calls transfer(fc, target) in FOUR code paths not just two, or (c) we keep the ThrowingLongSupplier as an elegant way to perform the 2x2 matrix. SelectableChannel is an abstract class, not an interface so you can't extend both FileChannel and SelectableChannel. So when the source channel is a FileChannel then it should only be concerned with whether the target channel is a SelectableChannel or not. I think we can keep it very simple, along the lines of the method I pasted in above. If you choose to include the case where the target is a SelectableChannel (and I have no objection to doing that) then the test needs to exercise that code. This means the test needs an input stream created on a source file/FileChannel, and an output stream created an channel that is WritableByteChannel & SelectableChannel. A Pipe.SinkChannel or a connected SocketChannel will do fine. If you choose to leave out selectable channels for the first PR then file -> file is okay, which I think is what your current test is doing. ------------- PR: https://git.openjdk.java.net/jdk/pull/5179 From bpb at openjdk.java.net Thu Aug 19 20:47:28 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 19 Aug 2021 20:47:28 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v2] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: <3zRStl6gMWV-fqHKYyBhEG2xJ1X9iaKHf1FDmVuUkW8=.1abb8df7-f5ab-48bb-862f-bdc26179f67b@github.com> On Thu, 19 Aug 2021 01:06:47 GMT, Brian Burkhalter wrote: >> This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8265261: Fix the section which tests cancellation It looks like the duration threshold `DURATION_MAX_IN_MS` was added in the change which resolved [JDK-6993267](https://bugs.openjdk.java.net/browse/JDK-6993267). A comment in that issue is "We should improve the test to work on very fast machines." So the copy must have been too fast for it to be interrupted and without the duration threshold there were spurious failures. `Thread.interrupted()` in the main thread indeed seems unnecessary. I think a further look at the semantics, as suggested, is in order. It looks like the duration threshold `DURATION_MAX_IN_MS` was added in the change which resolved [JDK-6993267](https://bugs.openjdk.java.net/browse/JDK-6993267). A comment in that issue is "We should improve the test to work on very fast machines." So the copy must have been too fast for it to be interrupted and without the duration threshold there were spurious failures. `Thread.interrupted()` in the main thread indeed seems unnecessary. I think a further look at the semantics, as suggested, is in order. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From mik3hall at gmail.com Thu Aug 19 23:44:57 2021 From: mik3hall at gmail.com (Michael Hall) Date: Thu, 19 Aug 2021 18:44:57 -0500 Subject: DefaultFileSystemProvider Message-ID: <4D9C9EC6-3584-4042-A2FC-2B3491703E1B@gmail.com> The recent fix for JDK-8263940 that just became available with the jdk18 ea appears to resolve about all remaining issues for my application. I had some initial problems with this that seemed module related. I verified it worked with modular eliminated. I then noticed I was duplicating the provider code both in class path and module path. I had thought I just split out my Apple eio replacement and additions to a module. But the entire provider was there as well. Given that I tried to see if it would run with the provider running strictly as a modular addition. It did not. Caused by: java.lang.Error: java.lang.ClassNotFoundException: us.hall.trz.osx.MacFileSystemProvider Possibly something in my setup is wrong but if you are looking for complete coverage you might want to verify that running a class path app with a default provider included as modular works. I am having some difficulty at the moment doing the split of class path jar provider and Apple eio replacement code to modular. But I am fairly certain I can run the app class path with modular as long as the provider isn?t included with the modular. So good enough for my purposes. From bpb at openjdk.java.net Fri Aug 20 01:28:46 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Fri, 20 Aug 2021 01:28:46 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v2] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: <9p5mpYMRzM9qIzHyRcDOciXBZjmEDrrd5ONe8sxRpkU=.bdb5a595-945b-4af7-870f-9f52621d182f@github.com> On Thu, 19 Aug 2021 01:06:47 GMT, Brian Burkhalter wrote: >> This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8265261: Fix the section which tests cancellation This latest version might be overthinking it, but I doubled the size of the test file created, and changed the interrupt and cancellation sections to make multiple attempts to interrupt the copy. I ran 80 iterations on each of the pertinent platforms with no failures. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Fri Aug 20 01:28:45 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Fri, 20 Aug 2021 01:28:45 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v3] In-Reply-To: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: > This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8265261: Make multiple attempts to interrupt and cancel ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5154/files - new: https://git.openjdk.java.net/jdk/pull/5154/files/85f78867..c02a8d15 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5154&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5154&range=01-02 Stats: 57 lines in 1 file changed: 27 ins; 5 del; 25 mod Patch: https://git.openjdk.java.net/jdk/pull/5154.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5154/head:pull/5154 PR: https://git.openjdk.java.net/jdk/pull/5154 From github.com+1701815+mkarg at openjdk.java.net Fri Aug 20 06:22:27 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Fri, 20 Aug 2021 06:22:27 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() In-Reply-To: References: <4aYyUY5raNV7PBhtlbPL9Mrqaoy4eeLct4uoq4jI1KI=.500ab3f2-1981-4a4d-8ea5-52da621f3a4d@github.com> Message-ID: On Thu, 19 Aug 2021 18:25:39 GMT, Alan Bateman wrote: > SelectableChannel is an abstract class, not an interface so you can't extend both FileChannel and SelectableChannel. So when the source channel is a FileChannel then it should only be concerned with whether the target channel is a SelectableChannel or not. I think we can keep it very simple, along the lines of the method I pasted in above. Silly me, you are right, it is an abstract class! Totally missed this! > If you choose to include the case where the target is a SelectableChannel (and I have no objection to doing that) then the test needs to exercise that code. This means the test needs an input stream created on a source file/FileChannel, and an output stream created an channel that is WritableByteChannel & SelectableChannel. A Pipe.SinkChannel or a connected SocketChannel will do fine. If you choose to leave out selectable channels for the first PR then file -> file is okay, which I think is what your current test is doing. Thanks a lot for the detailed explanation! I now better understood your concerns and will change the PR accordingly! ------------- PR: https://git.openjdk.java.net/jdk/pull/5179 From dfuchs at openjdk.java.net Fri Aug 20 08:55:29 2021 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Fri, 20 Aug 2021 08:55:29 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v3] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Fri, 20 Aug 2021 01:28:45 GMT, Brian Burkhalter wrote: >> This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8265261: Make multiple attempts to interrupt and cancel test/jdk/java/nio/file/Files/InterruptCopy.java line 92: > 90: // at a fixed rate after a delay > 91: final Thread me = Thread.currentThread(); > 92: Future wakeup = pool.scheduleAtFixedRate(new Runnable() { Is it really a good idea? The interrupted state should stick to the thread until it is cleared. It should not really matter if you interrupt the thread too early - but if you interrupt the thread *again* after the File copy was interrupted, won't it skew the logic in the test? Especially WRT double checking that the interrupt status was/wasn't cleared? Also shouldn't you cancel this wakeup at some point before testing cancel below? ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Fri Aug 20 15:44:32 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Fri, 20 Aug 2021 15:44:32 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v3] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Fri, 20 Aug 2021 08:51:23 GMT, Daniel Fuchs wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8265261: Make multiple attempts to interrupt and cancel > > test/jdk/java/nio/file/Files/InterruptCopy.java line 92: > >> 90: // at a fixed rate after a delay >> 91: final Thread me = Thread.currentThread(); >> 92: Future wakeup = pool.scheduleAtFixedRate(new Runnable() { > > Is it really a good idea? The interrupted state should stick to the thread until it is cleared. It should not really matter if you interrupt the thread too early - but if you interrupt the thread *again* after the File copy was interrupted, won't it skew the logic in the test? Especially WRT double checking that the interrupt status was/wasn't cleared? Also shouldn't you cancel this wakeup at some point before testing cancel below? I was not sure about the fixed rate firing here so you might be right. The wakeup is cancelled by `wakeup.get()` unless I misunderstand. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Fri Aug 20 15:44:32 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Fri, 20 Aug 2021 15:44:32 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v3] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Fri, 20 Aug 2021 15:39:21 GMT, Brian Burkhalter wrote: >> test/jdk/java/nio/file/Files/InterruptCopy.java line 92: >> >>> 90: // at a fixed rate after a delay >>> 91: final Thread me = Thread.currentThread(); >>> 92: Future wakeup = pool.scheduleAtFixedRate(new Runnable() { >> >> Is it really a good idea? The interrupted state should stick to the thread until it is cleared. It should not really matter if you interrupt the thread too early - but if you interrupt the thread *again* after the File copy was interrupted, won't it skew the logic in the test? Especially WRT double checking that the interrupt status was/wasn't cleared? Also shouldn't you cancel this wakeup at some point before testing cancel below? > > I was not sure about the fixed rate firing here so you might be right. The wakeup is cancelled by `wakeup.get()` unless I misunderstand. Correction: I think an explicit `wakeup.cancel()` is needed. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Fri Aug 20 17:39:46 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Fri, 20 Aug 2021 17:39:46 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v4] In-Reply-To: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: <07btl0RTowKHuiME7AYCgkJtO8d8fhZ52Y1g99lA5xY=.cbb17bb0-dc29-4c70-bfd5-ac529e8c9e3d@github.com> > This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8265261: Drop countdown latch and repeats of interrupts ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5154/files - new: https://git.openjdk.java.net/jdk/pull/5154/files/c02a8d15..752eb4b6 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5154&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5154&range=02-03 Stats: 42 lines in 1 file changed: 1 ins; 25 del; 16 mod Patch: https://git.openjdk.java.net/jdk/pull/5154.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5154/head:pull/5154 PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Sat Aug 21 00:05:26 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Sat, 21 Aug 2021 00:05:26 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v4] In-Reply-To: <07btl0RTowKHuiME7AYCgkJtO8d8fhZ52Y1g99lA5xY=.cbb17bb0-dc29-4c70-bfd5-ac529e8c9e3d@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> <07btl0RTowKHuiME7AYCgkJtO8d8fhZ52Y1g99lA5xY=.cbb17bb0-dc29-4c70-bfd5-ac529e8c9e3d@github.com> Message-ID: On Fri, 20 Aug 2021 17:39:46 GMT, Brian Burkhalter wrote: >> This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8265261: Drop countdown latch and repeats of interrupts I have another version in progress and it would be worth waiting to review that one once it is published. The new version succeeded in 539 of 540 runs on four platforms (135 runs per platform). The failed case printed this information: ```Creating source file... Source file created. Copying file at 1629496891924ms... Interrupting at 1629496891934ms... Done copying at 1629496892261ms... ----------System.err:(13/761)---------- java.lang.RuntimeException: Copy was not interrupted at InterruptCopy.doTest(InterruptCopy.java:110) at InterruptCopy.main(InterruptCopy.java:60) The interrupt call was made 10ms after the copy started, but the copy nonetheless ran another 327ms somehow without catching the interrupt. At present this is baffling. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From github.com+1701815+mkarg at openjdk.java.net Sat Aug 21 14:44:50 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 21 Aug 2021 14:44:50 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) Message-ID: As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). ------------- Commit messages: - 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) Changes: https://git.openjdk.java.net/jdk/pull/5209/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8265891 Stats: 433 lines in 4 files changed: 353 ins; 76 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/5209.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5209/head:pull/5209 PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Sat Aug 21 14:44:50 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 21 Aug 2021 14:44:50 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) In-Reply-To: References: Message-ID: On Sat, 21 Aug 2021 14:33:43 GMT, Markus KARG wrote: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). @AlanBateman @bplb This PR is a spin-off for the single case of FileChannel->FileChannel only, as suggested by you. I kindly request your review. Thanks. ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Sat Aug 21 14:46:22 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 21 Aug 2021 14:46:22 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() In-Reply-To: References: <4aYyUY5raNV7PBhtlbPL9Mrqaoy4eeLct4uoq4jI1KI=.500ab3f2-1981-4a4d-8ea5-52da621f3a4d@github.com> Message-ID: <4zyHg8isi-8lWUm9nwvT1VTJdWVlQICqT1tQQ8R6N7k=.0b6dcaa2-36ec-45a9-a882-166687ee3c04@github.com> On Thu, 19 Aug 2021 18:25:39 GMT, Alan Bateman wrote: >>> I had hoped that ThrowingLongSupplier would go away. If you keep your 2-arg transfer method then we should be able to implementation transferTo very simply, something like >>> >>> ``` >>> public long transferTo(OutputStream out) throws IOException { >>> if (ch instanceof FileChannel fc >>> && out instanceof ChannelOutputStream cos) { >>> WritableByteChannel target = cos.channel(); >>> if (target instanceof SelectableChannel sc) { >>> synchronized (sc.blockingLock()) { >>> if (!sc.isBlocking()) >>> throw new IllegalBlockingModeException(); >>> return transfer(fc, target); >>> } >>> } >>> return transfer(fc, target); >>> } >>> return super.transferTo(out); >>> } >>> ``` >> >> This is not true. It will only check the blocking mode for the target. But technically it is possible that someone writes an implementation of FileChannel which implements SelectableChannel. In that case it is needed to check the blocking mode on the source, too. So either we (a) ignore this possibility, (b) write a very complex code which calls transfer(fc, target) in FOUR code paths not just two, or (c) we keep the ThrowingLongSupplier as an elegant way to perform the 2x2 matrix. > >> This is not true. It will only check the blocking mode for the target. But technically it is possible that someone writes an implementation of FileChannel which implements SelectableChannel. In that case it is needed to check the blocking mode on the source, too. So either we (a) ignore this possibility, (b) write a very complex code which calls transfer(fc, target) in FOUR code paths not just two, or (c) we keep the ThrowingLongSupplier as an elegant way to perform the 2x2 matrix. > > SelectableChannel is an abstract class, not an interface so you can't extend both FileChannel and SelectableChannel. So when the source channel is a FileChannel then it should only be concerned with whether the target channel is a SelectableChannel or not. I think we can keep it very simple, along the lines of the method I pasted in above. > > If you choose to include the case where the target is a SelectableChannel (and I have no objection to doing that) then the test needs to exercise that code. This means the test needs an input stream created on a source file/FileChannel, and an output stream created an channel that is WritableByteChannel & SelectableChannel. A Pipe.SinkChannel or a connected SocketChannel will do fine. If you choose to leave out selectable channels for the first PR then file -> file is okay, which I think is what your current test is doing. @AlanBateman I have pushed another PR https://github.com/openjdk/jdk/pull/5209 that handles only the case FileChannel->FileChannel, so we will have some progress there and once *that* one is merged I will come back to the FileChannel->SelectableChannel case later. ------------- PR: https://git.openjdk.java.net/jdk/pull/5179 From github.com+1701815+mkarg at openjdk.java.net Sat Aug 21 18:08:58 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 21 Aug 2021 18:08:58 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() [v2] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains two new commits since the last revision: - 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() As suggested in 8265891 this change implements a performance optimization for ChannelInputStream.transferTo(): It uses FileChannel.transferTo() in case the wrapped channel actually is a FileChannel, as in that case, the actual work of transferring bytes can potentially get offloaded to the operating system / file system. As more such optimizations are likely to follow in the very same code segment, this change is deliberately elaborate and rather complex in its design, e. g. it already holds blocking locks for the complete time of operation on input and output channels. Signed-off-by: Markus Karg - 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) As suggested in 8265891 this change implements a performance optimization for ChannelInputStream.transferTo(): It uses FileChannel.transferTo() in case both wrapped channels actually are FileChannels, as in that case, the actual work of transferring bytes can potentially get offloaded to the operating system / file system. Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5179/files - new: https://git.openjdk.java.net/jdk/pull/5179/files/84a7b06c..f03ca97d Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=00-01 Stats: 46 lines in 2 files changed: 24 ins; 12 del; 10 mod Patch: https://git.openjdk.java.net/jdk/pull/5179.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5179/head:pull/5179 PR: https://git.openjdk.java.net/jdk/pull/5179 From bpb at openjdk.java.net Mon Aug 23 18:06:34 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Mon, 23 Aug 2021 18:06:34 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v4] In-Reply-To: <07btl0RTowKHuiME7AYCgkJtO8d8fhZ52Y1g99lA5xY=.cbb17bb0-dc29-4c70-bfd5-ac529e8c9e3d@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> <07btl0RTowKHuiME7AYCgkJtO8d8fhZ52Y1g99lA5xY=.cbb17bb0-dc29-4c70-bfd5-ac529e8c9e3d@github.com> Message-ID: On Fri, 20 Aug 2021 17:39:46 GMT, Brian Burkhalter wrote: >> This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8265261: Drop countdown latch and repeats of interrupts I tested in my work in progress by delaying the sleep until 10ms after the interrupt was issued and the copy was not interrupted. I expect that this is the reason for the 1 failure out of 540 executions mentioned above. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Mon Aug 23 18:06:35 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Mon, 23 Aug 2021 18:06:35 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v3] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Fri, 20 Aug 2021 15:41:33 GMT, Brian Burkhalter wrote: >> I was not sure about the fixed rate firing here so you might be right. The wakeup is cancelled by `wakeup.get()` unless I misunderstand. > > Correction: I think an explicit `wakeup.cancel()` is needed. Apparently setting the interrupted state prior to the copy does _not_ cause the copy to be interrupted. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Tue Aug 24 00:30:48 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 24 Aug 2021 00:30:48 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v5] In-Reply-To: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: > This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8265261: Reinstate duration check and latches but with countdown 1 ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5154/files - new: https://git.openjdk.java.net/jdk/pull/5154/files/752eb4b6..4ba51342 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5154&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5154&range=03-04 Stats: 33 lines in 1 file changed: 16 ins; 4 del; 13 mod Patch: https://git.openjdk.java.net/jdk/pull/5154.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5154/head:pull/5154 PR: https://git.openjdk.java.net/jdk/pull/5154 From bpb at openjdk.java.net Tue Aug 24 00:34:25 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 24 Aug 2021 00:34:25 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v5] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Tue, 24 Aug 2021 00:30:48 GMT, Brian Burkhalter wrote: >> This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8265261: Reinstate duration check and latches but with countdown 1 Version 04 revives the use of `CountDownLatch` but with a count of `1` instead of `2`. It is used for both the interrupt and the cancel cases to make the interrupting thread wait until just before `copy()` before the interrupting thread starts its sleep delay. For the interrupt case, a minimum duration for copying leading to a test failure is reinstated. This tries to prevent a false failure if the copy was much faster than expected. ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From github.com+77050469+pirateskipper at openjdk.java.net Tue Aug 24 09:41:24 2021 From: github.com+77050469+pirateskipper at openjdk.java.net (pirateskipper) Date: Tue, 24 Aug 2021 09:41:24 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) In-Reply-To: References: Message-ID: On Sat, 21 Aug 2021 14:33:43 GMT, Markus KARG wrote: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). test/jdk/sun/nio/ch/ChannelInputStream/TransferTo.java line 181: > 179: > 180: public static void assertThrows(Thrower thrower, Class throwable, String message) { > 181: Throwable thrown; ` Throwable thrown = null; try { thrower.run();` ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From dfuchs at openjdk.java.net Tue Aug 24 11:04:30 2021 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Tue, 24 Aug 2021 11:04:30 GMT Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v5] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Tue, 24 Aug 2021 00:30:48 GMT, Brian Burkhalter wrote: >> This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8265261: Reinstate duration check and latches but with countdown 1 If the test campaign shows better results with this latter version then I have no objection: it looks better to me than the previous repeated interrupt logic. ------------- Marked as reviewed by dfuchs (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/5154 From alanb at openjdk.java.net Tue Aug 24 12:20:24 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Tue, 24 Aug 2021 12:20:24 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) In-Reply-To: References: Message-ID: On Sat, 21 Aug 2021 14:33:43 GMT, Markus KARG wrote: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). There are 3 PRs open, I'll ignore #4263 and #5179 for now. The moving of the OutputStream implementation from Channels to sun.nio.ch.ChannelOutputStream looks fine. In the class description it should be "accessible" as it visible. You can drop "but not be part of the java.base module API" too. Changing the bounds check to use Objects.checkFromIndexSize looks fine. It would be good if you could re-format 1-arg transferTo to something like this: if (out instanceof ChannelOutputStream cos && ch instanceof FileChannel fc && cos.channel() instanceof FileChannel dst) { } just to to avoid the really long line as that gets annoying when looking at side-by-side diffs. The 2-arg transferTo methods look okay. An alternative, that avoids the need for "srcPos + bytesWritten" in 3 places is to just something like: long initialPos = src.position(); long pos = initialPos; try { while (pos < src.size()) { pos += src.transferTo(pos, Long.MAX_VALUE, dst); } } finally { src.position(pos); } return pos - initialPos; @bplb You looked at the test in the first PR, have you look at this one? I suspect it would be a lot cleaner/maintainable if it were written as a TestNG test. I assume the transfer loop never has more than one iteration because the file sizes are <2GB. If it were writing this test then I would probably use Files/Arrays.mismatch to compare the contents. ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From brian.burkhalter at oracle.com Tue Aug 24 16:44:17 2021 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 24 Aug 2021 16:44:17 +0000 Subject: RFR: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted [v5] In-Reply-To: References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: <99FF7705-4A02-424D-BD16-AF7DFE42D447@oracle.com> On Aug 24, 2021, at 4:04 AM, Daniel Fuchs > wrote: On Tue, 24 Aug 2021 00:30:48 GMT, Brian Burkhalter > wrote: This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8265261: Reinstate duration check and latches but with countdown 1 If the test campaign shows better results with this latter version then I have no objection: it looks better to me than the previous repeated interrupt logic. This does show better results. I am not however convinced that it is better than the previously posted fixed rate interrupt version. I verified that if the interrupt is issued before the copy begins then it is not recognized by `copy()`. Given that, repeating might give a more predictable result. I would agree though to go with this version and revisit it once more should failures be seen. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpb at openjdk.java.net Tue Aug 24 17:08:36 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 24 Aug 2021 17:08:36 GMT Subject: Integrated: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> References: <_Q0YB0kAMwBVfCw7PnaiaGi1X3zkzLCHlZZurKmNya8=.a823bd06-14ef-4980-b9b9-535f92f22c1c@github.com> Message-ID: On Wed, 18 Aug 2021 02:32:53 GMT, Brian Burkhalter wrote: > This proposal suggests to change the timing of testing whether a file copy is terminated by an interrupt. This pull request has now been integrated. Changeset: aaedac63 Author: Brian Burkhalter URL: https://git.openjdk.java.net/jdk/commit/aaedac635a0a7ca1f73dcf85336230cc537fcdf2 Stats: 60 lines in 1 file changed: 39 ins; 0 del; 21 mod 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted Reviewed-by: dfuchs ------------- PR: https://git.openjdk.java.net/jdk/pull/5154 From github.com+1701815+mkarg at openjdk.java.net Tue Aug 24 20:11:56 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Tue, 24 Aug 2021 20:11:56 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() [v3] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has updated the pull request incrementally with one additional commit since the last revision: Improved class description of ChannelOutputStream Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5179/files - new: https://git.openjdk.java.net/jdk/pull/5179/files/f03ca97d..17b43e6c Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=01-02 Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/5179.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5179/head:pull/5179 PR: https://git.openjdk.java.net/jdk/pull/5179 From github.com+1701815+mkarg at openjdk.java.net Tue Aug 24 20:24:00 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Tue, 24 Aug 2021 20:24:00 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v2] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has updated the pull request incrementally with one additional commit since the last revision: Improved class description of ChannelOutputStream Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5209/files - new: https://git.openjdk.java.net/jdk/pull/5209/files/bed6daf8..f35cd7d7 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=00-01 Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/5209.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5209/head:pull/5209 PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Tue Aug 24 20:31:04 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Tue, 24 Aug 2021 20:31:04 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v3] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has updated the pull request incrementally with one additional commit since the last revision: Reformatted to prevent long line Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5209/files - new: https://git.openjdk.java.net/jdk/pull/5209/files/f35cd7d7..33bef3d6 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=01-02 Stats: 3 lines in 1 file changed: 2 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/5209.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5209/head:pull/5209 PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Tue Aug 24 20:40:32 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Tue, 24 Aug 2021 20:40:32 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) In-Reply-To: References: Message-ID: On Tue, 24 Aug 2021 12:17:18 GMT, Alan Bateman wrote: > There are 3 PRs open, I'll ignore #4263 and #5179 for now. Yes please ignore them for now. The other two PRs are suspendend as they cover bigger pictures and will be resumed by me *later*. > The moving of the OutputStream implementation from Channels to sun.nio.ch.ChannelOutputStream looks fine. In the class description it should be "accessible" as it visible. You can drop "but not be part of the java.base module API" too. Done in https://github.com/openjdk/jdk/pull/5209/commits/f35cd7d74f2ac15f60e7d97395ec71deefeafb76. > It would be good if you could re-format 1-arg transferTo to something like this: > ... > just to to avoid the really long line as that gets annoying when looking at side-by-side diffs. Never had this problem as I'm using inline-diff tools only, but done in https://github.com/openjdk/jdk/pull/5209/commits/33bef3d6ef33bb9eddc66e69c1a1e6fd9d565f34. > An alternative, that avoids the need for "srcPos + bytesWritten" in 3 places is to just something like: > ... Does that mean that I must change it? As the method is mostly I/O bound I do not see what we actually would gain. > I suspect it would be a lot cleaner/maintainable if it were written as a TestNG test. I assume the transfer loop never has more than one iteration because the file sizes are <2GB. If it were writing this test then I would probably use Files/Arrays.mismatch to compare the contents. Understood, but please note that this code actually was simply copied from the original tests that Brian provided as a blue-print, which neither uses TestNG nor Files/Arrays.mismatch nor tests larger files. So if you ask me to change that, in turn someone should change the original tests then to keep the code in sync. So can we keep the tests as-is or am I forced to use TestNG and File/Arrays.mismatch? ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Tue Aug 24 20:55:05 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Tue, 24 Aug 2021 20:55:05 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v4] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has updated the pull request incrementally with one additional commit since the last revision: Simplified copy loop Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5209/files - new: https://git.openjdk.java.net/jdk/pull/5209/files/33bef3d6..205b9b21 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=02-03 Stats: 7 lines in 1 file changed: 1 ins; 1 del; 5 mod Patch: https://git.openjdk.java.net/jdk/pull/5209.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5209/head:pull/5209 PR: https://git.openjdk.java.net/jdk/pull/5209 From bpb at openjdk.java.net Tue Aug 24 22:15:28 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 24 Aug 2021 22:15:28 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) In-Reply-To: References: Message-ID: On Tue, 24 Aug 2021 12:17:18 GMT, Alan Bateman wrote: >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. >> >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). > > There are 3 PRs open, I'll ignore #4263 and #5179 for now. > > The moving of the OutputStream implementation from Channels to sun.nio.ch.ChannelOutputStream looks fine. In the class description it should be "accessible" as it visible. You can drop "but not be part of the java.base module API" too. > > Changing the bounds check to use Objects.checkFromIndexSize looks fine. > > It would be good if you could re-format 1-arg transferTo to something like this: > > if (out instanceof ChannelOutputStream cos > && ch instanceof FileChannel fc > && cos.channel() instanceof FileChannel dst) { > > } > > just to to avoid the really long line as that gets annoying when looking at side-by-side diffs. > > The 2-arg transferTo methods look okay. An alternative, that avoids the need for "srcPos + bytesWritten" in 3 places is to just something like: > > long initialPos = src.position(); > long pos = initialPos; > try { > while (pos < src.size()) { > pos += src.transferTo(pos, Long.MAX_VALUE, dst); > } > } finally { > src.position(pos); > } > return pos - initialPos; > > > @bplb You looked at the test in the first PR, have you look at this one? I suspect it would be a lot cleaner/maintainable if it were written as a TestNG test. I assume the transfer loop never has more than one iteration because the file sizes are <2GB. If it were writing this test then I would probably use Files/Arrays.mismatch to compare the contents. @AlanBateman It does not look like this test is substantively different than the one in the first PR aside from constraining the cases tested. One thing I think is insufficient is only dealing with streams at position zero. In the test I recently added, [java/io/FileInputStream/TransferTo.java](https://github.com/openjdk/jdk/blob/master/test/jdk/java/io/FileInputStream/TransferTo.java), I included random initial positions for both input and output. For random positions it is better to use `jdk.test.lib.RandomFactory.getRandom()` instead of `new Random()` as this will print the seed value which can be useful in reproducing any problems. (For data content alone, `new Random()` is fine because we don't care what those bytes are.) ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From alanb at openjdk.java.net Wed Aug 25 11:19:25 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Wed, 25 Aug 2021 11:19:25 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v4] In-Reply-To: References: Message-ID: <87UHudqL9BB6QcPC-3--ZOQjEX7TwzoeXST4i1jIvdA=.a80cff68-193e-4b4f-9eba-49c5578bdf35@github.com> On Tue, 24 Aug 2021 20:55:05 GMT, Markus KARG wrote: >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. >> >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). > > Markus KARG has updated the pull request incrementally with one additional commit since the last revision: > > Simplified copy loop > > Signed-off-by: Markus Karg > > There are 3 PRs open, I'll ignore #4263 and #5179 for now. > > Yes please ignore them for now. The other two PRs are suspendend as they cover bigger pictures and will be resumed by me _later_. Okay. The implementation changes in the current patch (205b9b21) look good. I think we will need to iterate on the test. ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Wed Aug 25 16:21:33 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Wed, 25 Aug 2021 16:21:33 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v4] In-Reply-To: References: Message-ID: On Tue, 24 Aug 2021 20:55:05 GMT, Markus KARG wrote: >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. >> >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). > > Markus KARG has updated the pull request incrementally with one additional commit since the last revision: > > Simplified copy loop > > Signed-off-by: Markus Karg > > > There are 3 PRs open, I'll ignore #4263 and #5179 for now. > > > > > > Yes please ignore them for now. The other two PRs are suspendend as they cover bigger pictures and will be resumed by me _later_. > > Okay. The implementation changes in the current patch ([205b9b2](https://github.com/openjdk/jdk/commit/205b9b21e003937f91311eb91d7671e0810684bb)) look good. I think we will need to iterate on the test. Ok so I will do as Brian said and include a test case with random initial positions. Note that my test will do that with the default case, too, so maybe someone should do that for the original test of the default implementation, too, to stay in sync. ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From bpb at openjdk.java.net Wed Aug 25 21:40:22 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 25 Aug 2021 21:40:22 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v4] In-Reply-To: References: Message-ID: On Tue, 24 Aug 2021 20:55:05 GMT, Markus KARG wrote: >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. >> >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). > > Markus KARG has updated the pull request incrementally with one additional commit since the last revision: > > Simplified copy loop > > Signed-off-by: Markus Karg Why not just use [java/io/FileInputStream/TransferTo.java](https://github.com/openjdk/jdk/blob/master/test/jdk/java/io/FileInputStream/TransferTo.java) as a starting point? ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From bpb at openjdk.java.net Wed Aug 25 23:43:38 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 25 Aug 2021 23:43:38 GMT Subject: RFR: 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted Message-ID: In the interrupt case, if the copy does not throw an `IOException`, rather than using the copy duration versus a threshold as the criterion for failure, instead check whether the target file does **not** exist, and if it does not, then make the test fail as this would indicate that the copy was in fact interrupted but did not throw an exception in response. ------------- Commit messages: - 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted Changes: https://git.openjdk.java.net/jdk/pull/5260/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5260&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8272964 Stats: 10 lines in 1 file changed: 7 ins; 1 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/5260.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5260/head:pull/5260 PR: https://git.openjdk.java.net/jdk/pull/5260 From bpb at openjdk.java.net Wed Aug 25 23:43:38 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 25 Aug 2021 23:43:38 GMT Subject: RFR: 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: References: Message-ID: On Wed, 25 Aug 2021 23:37:50 GMT, Brian Burkhalter wrote: > In the interrupt case, if the copy does not throw an `IOException`, rather than using the copy duration versus a threshold as the criterion for failure, instead check whether the target file does **not** exist, and if it does not, then make the test fail as this would indicate that the copy was in fact interrupted but did not throw an exception in response. Before this PR was created, this test had been run 700 times on each of four different platforms without any failures. ------------- PR: https://git.openjdk.java.net/jdk/pull/5260 From msheppar at openjdk.java.net Thu Aug 26 00:56:27 2021 From: msheppar at openjdk.java.net (Mark Sheppard) Date: Thu, 26 Aug 2021 00:56:27 GMT Subject: RFR: 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: References: Message-ID: On Wed, 25 Aug 2021 23:37:50 GMT, Brian Burkhalter wrote: > In the interrupt case, if the copy does not throw an `IOException`, rather than using the copy duration versus a threshold as the criterion for failure, instead check whether the target file does **not** exist, and if it does not, then make the test fail as this would indicate that the copy was in fact interrupted but did not throw an exception in response. That's look like a good approach, because the duration threshold test is always subject to the load and the number of extant threads in the test system ... for recent failures the number of threads is significant (> 1000) Also, when following the copy calls flows there is significant amount of processing prior to the actual copy activity taking place and that occurs in Cancellable in a newly launched "copying thread". What happens if the copy thread has yet to run? Additionally, would it be useful for the copy interrupter thread to check for the existence of the target file, prior to invoking the interrupt e.g. while (Files.notExist(target)) Thread.sleep(...); ------------- PR: https://git.openjdk.java.net/jdk/pull/5260 From bpb at openjdk.java.net Thu Aug 26 02:56:27 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 26 Aug 2021 02:56:27 GMT Subject: RFR: 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: References: Message-ID: On Wed, 25 Aug 2021 23:37:50 GMT, Brian Burkhalter wrote: > In the interrupt case, if the copy does not throw an `IOException`, rather than using the copy duration versus a threshold as the criterion for failure, instead check whether the target file does **not** exist, and if it does not, then make the test fail as this would indicate that the copy was in fact interrupted but did not throw an exception in response. My testing showed that if the interrupt is set on the main thread, the copy thread, before the copy actually starts, then the interrupt is missed and the copy runs to completion. As for the `Files.notExists(target)` loop, I tried that in the previous attempt to fix this issue and it appeared that on Windows this call is too slow to use in this context and does not fix the problem. ------------- PR: https://git.openjdk.java.net/jdk/pull/5260 From github.com+1701815+mkarg at openjdk.java.net Thu Aug 26 06:21:23 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 26 Aug 2021 06:21:23 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v4] In-Reply-To: References: Message-ID: On Wed, 25 Aug 2021 21:37:28 GMT, Brian Burkhalter wrote: > Why not just use [java/io/FileInputStream/TransferTo.java](https://github.com/openjdk/jdk/blob/master/test/jdk/java/io/FileInputStream/TransferTo.java) as a starting point? Because I already invested considerable time into my test already and in fact do like my test more due to the flexibility it provides for my future PRs that will follow soon (adding more transfer cases), and very much like it as it is, so if only a small change is actually needed here then I will add that change and we're done with this PR and I will go on to re-enable other test cases. ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From dfuchs at openjdk.java.net Thu Aug 26 09:53:26 2021 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Thu, 26 Aug 2021 09:53:26 GMT Subject: RFR: 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: References: Message-ID: <4MiEkjXAeKF9oaoP0VauFCJrg5QlyAnLUaD4Ff-_-Mk=.6a284cdd-1985-47f9-991a-3aa9ed333bfc@github.com> On Wed, 25 Aug 2021 23:37:50 GMT, Brian Burkhalter wrote: > In the interrupt case, if the copy does not throw an `IOException`, rather than using the copy duration versus a threshold as the criterion for failure, instead check whether the target file does **not** exist, and if it does not, then make the test fail as this would indicate that the copy was in fact interrupted but did not throw an exception in response. I like that! ------------- Marked as reviewed by dfuchs (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/5260 From msheppar at openjdk.java.net Thu Aug 26 19:27:24 2021 From: msheppar at openjdk.java.net (Mark Sheppard) Date: Thu, 26 Aug 2021 19:27:24 GMT Subject: RFR: 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: References: Message-ID: On Wed, 25 Aug 2021 23:37:50 GMT, Brian Burkhalter wrote: > In the interrupt case, if the copy does not throw an `IOException`, rather than using the copy duration versus a threshold as the criterion for failure, instead check whether the target file does **not** exist, and if it does not, then make the test fail as this would indicate that the copy was in fact interrupted but did not throw an exception in response. the solution looks hunky dory, but one question on the call flow in Unix for Files.copy: For the interruptible scenario, an auxiliary thread is used by Cancellable to process the copy. If the "main" thread associated with the Files.copy has yet to reach a call on the actual copy thread t.join when it is interrupted or the interrupt is delivered, then it looks like the interrupt will be missed and the InterruptExcpetion won't be generated. So is there a case for checking the interrupt status of the current thread, in the copy call flow, prior to invoking Cancellable.runInterruptibly, or before t.start for executing the actual copy or before t.join in the runInterruptibly method ? as such increasing the possibility of intercepting the Thread::interrupt in the test? So in the call flow if the current thread's interrupt status equals interrupted prior to invoking Cancellable.runInterruptibly then throw an IOException at that point, and additionally test the interrupt status prior to invoking t.start and t.join in runInterruptibly. The point I'm attempting to make is that the t.join in runInterruptibly is the juncture at which an interrupt can be caught, and prior to that, if I understand Thread::interrupt correctly, only the interrupt status will be set. As such, is the test failure not also alluding a potential flaw in the interruptibility of the File.copy? That there is the possibility of an interrupt being delivered prior to the actual invocation of the "real copy" ? ------------- PR: https://git.openjdk.java.net/jdk/pull/5260 From bpb at openjdk.java.net Thu Aug 26 20:13:26 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 26 Aug 2021 20:13:26 GMT Subject: RFR: 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: References: Message-ID: On Wed, 25 Aug 2021 23:37:50 GMT, Brian Burkhalter wrote: > In the interrupt case, if the copy does not throw an `IOException`, rather than using the copy duration versus a threshold as the criterion for failure, instead check whether the target file does **not** exist, and if it does not, then make the test fail as this would indicate that the copy was in fact interrupted but did not throw an exception in response. I tested three scenarios: 1. interrupt the thread which calls `Files.copy()` prior to calling `Files.copy()`; 2. self-interrupt the thread which calls `Files.copy()` in `Cancellable` on the line **before** `t.join()`. 3. self-interrupt the thread which calls `Files.copy()` in `Cancellable` on the line **after** `t.join()`. In the first two cases an `InterruptedException` was thrown. Therefore I don't believe that there is any need to check `isInterrupted()` to avoid missing an interrupt made before spawning the "NIO-Task" thread in which the `Cancellable` task is executed. The third case however reveals that if the thread which calls `Files.copy()` is interrupted in `runInterruptibly()` on the line _after_ `t.join()`, then the copy succeeds and no `InterruptedException` is thrown. So there is a small window in time between `t.join()` and the return from `Files.copy()` when the interrupt wil be ignored. ------------- PR: https://git.openjdk.java.net/jdk/pull/5260 From Alan.Bateman at oracle.com Fri Aug 27 07:08:57 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 27 Aug 2021 08:08:57 +0100 Subject: RFR: 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: References: Message-ID: <9f981b91-20e1-2ab4-321b-027d633dc9ef@oracle.com> On 26/08/2021 21:13, Brian Burkhalter wrote: > : > The third case however reveals that if the thread which calls `Files.copy()` is interrupted in `runInterruptibly()` on the line _after_ `t.join()`, then the copy succeeds and no `InterruptedException` is thrown. So there is a small window in time between `t.join()` and the return from `Files.copy()` when the interrupt wil be ignored. I don't immediately see an issue there. A thread can be interrupted at any time. If a thread in Files.copy is interrupted after the copy has completed but before the method returns then it just means that Files.copy will complete (successfully) but with the interrupt status set. -Alan. From myano at openjdk.java.net Fri Aug 27 08:46:42 2021 From: myano at openjdk.java.net (Masanori Yano) Date: Fri, 27 Aug 2021 08:46:42 GMT Subject: RFR: 8238274: (sctp) JDK-7118373 is not fixed for SctpChannel Message-ID: Please review this change to the Unix implementations of sun.nio.ch.sctp.Sctp*ChannelImpl#implCloseSelectableChannel() to be same as SocketChannelImpl at JDK-7118373. (The preClose is missing a check for the ST_KILLED state.) ------------- Commit messages: - 8238274: (sctp) JDK-7118373 is not fixed for SctpChannel Changes: https://git.openjdk.java.net/jdk/pull/5274/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5274&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8238274 Stats: 213 lines in 4 files changed: 210 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/5274.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5274/head:pull/5274 PR: https://git.openjdk.java.net/jdk/pull/5274 From bpb at openjdk.java.net Fri Aug 27 16:11:35 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Fri, 27 Aug 2021 16:11:35 GMT Subject: Integrated: 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted In-Reply-To: References: Message-ID: On Wed, 25 Aug 2021 23:37:50 GMT, Brian Burkhalter wrote: > In the interrupt case, if the copy does not throw an `IOException`, rather than using the copy duration versus a threshold as the criterion for failure, instead check whether the target file does **not** exist, and if it does not, then make the test fail as this would indicate that the copy was in fact interrupted but did not throw an exception in response. This pull request has now been integrated. Changeset: dfeb4132 Author: Brian Burkhalter URL: https://git.openjdk.java.net/jdk/commit/dfeb4132e402c0466740a029c3b1d2d213955822 Stats: 10 lines in 1 file changed: 7 ins; 1 del; 2 mod 8272964: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted Reviewed-by: dfuchs ------------- PR: https://git.openjdk.java.net/jdk/pull/5260 From alanb at openjdk.java.net Sat Aug 28 07:21:24 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Sat, 28 Aug 2021 07:21:24 GMT Subject: RFR: 8238274: (sctp) JDK-7118373 is not fixed for SctpChannel In-Reply-To: References: Message-ID: On Fri, 27 Aug 2021 08:37:46 GMT, Masanori Yano wrote: > Please review this change to the Unix implementations of sun.nio.ch.sctp.Sctp*ChannelImpl#implCloseSelectableChannel() > to be same as SocketChannelImpl at JDK-7118373. (The preClose is missing a check for the ST_KILLED state.) As the equivalent of JDK-7118373 in JDK 8 then the proposed change looks okay. At some point we may have to look at aligning the SCTP channel implementations where close has been significantly re-implemented. ------------- PR: https://git.openjdk.java.net/jdk/pull/5274 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 29 08:25:38 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 29 Aug 2021 08:25:38 GMT Subject: RFR: JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v5] In-Reply-To: References: Message-ID: <2KQq_ShHcllXANPQql6N45dS86ysQGzm6Qo3P7JPSSo=.49af059d-1ca0-4f24-9dda-631e71e9d8a3@github.com> > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has updated the pull request incrementally with one additional commit since the last revision: Testing random starting point within source Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5209/files - new: https://git.openjdk.java.net/jdk/pull/5209/files/205b9b21..e9163bf4 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=03-04 Stats: 36 lines in 1 file changed: 30 ins; 1 del; 5 mod Patch: https://git.openjdk.java.net/jdk/pull/5209.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5209/head:pull/5209 PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 29 11:41:38 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 29 Aug 2021 11:41:38 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v13] In-Reply-To: References: Message-ID: On Thu, 12 Aug 2021 01:10:15 GMT, Brian Burkhalter wrote: >> Markus KARG has updated the pull request incrementally with two additional commits since the last revision: >> >> - Draft: Eliminated duplicate code using lambda expressions >> - Draft: Use blocking mode also for target channel > > test/jdk/sun/nio/ch/ChannelInputStream/TransferTo.java line 93: > >> 91: checkTransferredContents(inputStreamProvider, outputStreamProvider, createRandomBytes(1024, 4096)); >> 92: // to span through several batches >> 93: checkTransferredContents(inputStreamProvider, outputStreamProvider, createRandomBytes(16384, 16384)); > > Should some random-sized buffers be tested? (Use `jdk.test.lib.RandomFactory` factory for this, not `j.u.Random`. The existing use of `Random` is fine.) > > Should some testing be done of streams with non-zero initial position? Done in https://github.com/openjdk/jdk/pull/5209, will rebase on that PR once merged. > test/jdk/sun/nio/ch/ChannelInputStream/TransferTo.java line 101: > >> 99: try (InputStream in = inputStreamProvider.input(inBytes); >> 100: OutputStream out = outputStreamProvider.output(recorder::set)) { >> 101: in.transferTo(out); > > The return value of `transferTo()` is not checked. Done in https://github.com/openjdk/jdk/pull/5209, will rebase on that PR once merged. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 29 11:44:55 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 29 Aug 2021 11:44:55 GMT Subject: RFR: JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v6] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: Testing random starting point within source and target Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5209/files - new: https://git.openjdk.java.net/jdk/pull/5209/files/e9163bf4..a043ae25 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=04-05 Stats: 13 lines in 1 file changed: 4 ins; 1 del; 8 mod Patch: https://git.openjdk.java.net/jdk/pull/5209.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5209/head:pull/5209 PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 29 12:18:28 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 29 Aug 2021 12:18:28 GMT Subject: RFR: JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v4] In-Reply-To: References: Message-ID: On Wed, 25 Aug 2021 21:37:28 GMT, Brian Burkhalter wrote: >> Markus KARG has updated the pull request incrementally with one additional commit since the last revision: >> >> Simplified copy loop >> >> Signed-off-by: Markus Karg > > Why not just use [java/io/FileInputStream/TransferTo.java](https://github.com/openjdk/jdk/blob/master/test/jdk/java/io/FileInputStream/TransferTo.java) as a starting point? @bplb @AlanBateman As discussed before, I have changed the test in this PR to cover the following cases: * Testing random buffer sizes * Testing random initial source position * Testing random initial target position I deliberately did not test the following: * 2GB+: IMHO makes not much sense to add such a heavy weight test as the expected benefit will not really outweigh the test complexity needed, will slow down test performance massively, and was not tested in the original implementation / in https://github.com/openjdk/jdk/blob/master/test/jdk/java/io/FileInputStream/TransferTo.java, too. There is no rule that we have to provide 100% test coverage in OpenJDK. Actually adding a test must be well justified and is not the only measure we have in the box to prevent the occurance of bugs in production: I suggest, instead of a 2GB+ test, we agree that the contributed implementation *looks* correct to each of us (= applying six-eyes principle). For such a simple code line, this should be enough QA, as in case it unexpectedly would break, it is very likely to get reported by beta testers long before GA (possibly by myself as many of my projects move around 4GB+ sized files in reality), which should be early enough to catch and fix. * Same for testing position-after-transfer. This is not covered by the original test, and due to our six eyes it is highly unlikely that it is broken or will break in future. So I'd say KISS here. Wherever it made sense (without sacrifying my intended modularity needed for other channel types tested later) I tried to follow the solutions found in https://github.com/openjdk/jdk/blob/master/test/jdk/java/io/FileInputStream/TransferTo.java. Thanks Bran, for this blue print. In fact I wonder if the introduction of 10 loops over random sizes actually will improve test qualify. My assumption is that it rather reduces test stability and test repeatability: In case there actually is a bug that happens only with a specific combination of buffer size and initial positions, ten iterations will be much too few to stably reproduce the bug at each execution of the test. More often that not, that ten loops will all pass without running into the problem given the huge max size. This will driver testers nuts! The improved entropy by introducing the JDK randomizer does not help much to improve this. Having said that, I copied the solution from Brian only because he came up with it, and I can li ve with the statistical unpredictability it induces, but just wanted to clearly say that I (spoiler: working for a company providing QA tools like FMEA and SPC software) would say introducing random here induces more harm due to non-repeatability than it actualy improves the detection rate of failures not detected by thorough six-eye review. ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 29 15:36:06 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 29 Aug 2021 15:36:06 GMT Subject: RFR: JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v7] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has updated the pull request incrementally with one additional commit since the last revision: fixed whitespace warning ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5209/files - new: https://git.openjdk.java.net/jdk/pull/5209/files/a043ae25..b80bf8ea Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=06 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=05-06 Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/5209.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5209/head:pull/5209 PR: https://git.openjdk.java.net/jdk/pull/5209 From alanb at openjdk.java.net Sun Aug 29 16:27:25 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Sun, 29 Aug 2021 16:27:25 GMT Subject: RFR: JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v4] In-Reply-To: References: Message-ID: On Sun, 29 Aug 2021 12:14:58 GMT, Markus KARG wrote: > * 2GB+: IMHO makes not much sense to add such a heavy weight test as the expected benefit will not really outweigh the test complexity needed, I'll try to get time to look at what you have in the next few days but just to say that the suggestion for exercising files > 2GB is because it's beyond the limit of the underlying sendfile. You should be able to check this quickly by printing the iteration count in ChannelInputStream.transferTo, it is always 1? ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 29 17:15:29 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 29 Aug 2021 17:15:29 GMT Subject: RFR: JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v6] In-Reply-To: References: Message-ID: On Sun, 29 Aug 2021 11:44:55 GMT, Markus KARG wrote: >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. >> >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). > > Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. > I'll try to get time to look at what you have in the next few days but just to say that the suggestion for exercising files > 2GB is because it's beyond the limit of the underlying sendfile. You should be able to check this quickly by printing the iteration count in ChannelInputStream.transferTo, it is always 1? I understand your concerns and can dispel them. I added a counter that prints the number of iterations and yes, just as you expected, due to the `Long.MAX_VALUE` there is only a single iteration. But then I simply reduced `Long.MAX_VALUE` to `1024` and the counter reported thousands of iterations -- just as expected. Even better, the test confirmed that still the number of reported bytes is still correct just as the file contents are. Hence in case sendfile *limits* the number of transferred bytes in production use, this quick check has proven that the code *is correct* and everything *actually* works fine. ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Sun Aug 29 17:15:30 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sun, 29 Aug 2021 17:15:30 GMT Subject: RFR: JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v7] In-Reply-To: References: Message-ID: On Tue, 24 Aug 2021 09:35:25 GMT, pirateskipper wrote: >> Markus KARG has updated the pull request incrementally with one additional commit since the last revision: >> >> fixed whitespace warning > > test/jdk/sun/nio/ch/ChannelInputStream/TransferTo.java line 181: > >> 179: >> 180: public static void assertThrows(Thrower thrower, Class throwable, String message) { >> 181: Throwable thrown; > > ` Throwable thrown = null; > try { > thrower.run();` Is that an OpenJDK best practice or why do you ask for this change? The code actually is a copy of existing OpenJDK code and not originally authored by me. ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Mon Aug 30 06:52:10 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Mon, 30 Aug 2021 06:52:10 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() [v4] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains six new commits since the last revision: - 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(WritableByteChannel) As suggested in 8265891 this change implements a performance optimization for ChannelInputStream.transferTo(): It uses FileChannel.transferTo(WritableByteChannel) in case the source channel actually is a FileChannel and the target channel is a WritableByteChannel, as in that case, the actual work of transferring bytes can potentially get offloaded to the operating system / file system. Signed-off-by: Markus Karg - fixed whitespace warning - Testing random starting point within source and target Signed-off-by: Markus Karg - Simplified copy loop Signed-off-by: Markus Karg - Reformatted to prevent long line Signed-off-by: Markus Karg - Improved class description of ChannelOutputStream Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5179/files - new: https://git.openjdk.java.net/jdk/pull/5179/files/17b43e6c..1e4c9deb Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=02-03 Stats: 51 lines in 2 files changed: 35 ins; 4 del; 12 mod Patch: https://git.openjdk.java.net/jdk/pull/5179.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5179/head:pull/5179 PR: https://git.openjdk.java.net/jdk/pull/5179 From github.com+1701815+mkarg at openjdk.java.net Mon Aug 30 13:52:57 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Mon, 30 Aug 2021 13:52:57 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() [v5] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(WritableByteChannel) As suggested in 8265891 this change implements a performance optimization for ChannelInputStream.transferTo(): It uses FileChannel.transferTo(WritableByteChannel) in case the source channel actually is a FileChannel and the target channel is a WritableByteChannel, as in that case, the actual work of transferring bytes can potentially get offloaded to the operating system / file system. Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5179/files - new: https://git.openjdk.java.net/jdk/pull/5179/files/1e4c9deb..56f16934 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=03-04 Stats: 211 lines in 1 file changed: 32 ins; 0 del; 179 mod Patch: https://git.openjdk.java.net/jdk/pull/5179.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5179/head:pull/5179 PR: https://git.openjdk.java.net/jdk/pull/5179 From github.com+1701815+mkarg at openjdk.java.net Mon Aug 30 14:02:06 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Mon, 30 Aug 2021 14:02:06 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() [v6] In-Reply-To: References: Message-ID: <181V3-aUkTGAOb4xCodIPnl_gF6IEu0GxQb6NiLbktc=.ca341c4a-8890-4ec4-8e30-ee47bf7f9700@github.com> > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(WritableByteChannel) As suggested in 8265891 this change implements a performance optimization for ChannelInputStream.transferTo(): It uses FileChannel.transferTo(WritableByteChannel) in case the source channel actually is a FileChannel and the target channel is a WritableByteChannel, as in that case, the actual work of transferring bytes can potentially get offloaded to the operating system / file system. Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5179/files - new: https://git.openjdk.java.net/jdk/pull/5179/files/56f16934..041e02fc Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=04-05 Stats: 212 lines in 1 file changed: 6 ins; 0 del; 206 mod Patch: https://git.openjdk.java.net/jdk/pull/5179.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5179/head:pull/5179 PR: https://git.openjdk.java.net/jdk/pull/5179 From github.com+1701815+mkarg at openjdk.java.net Mon Aug 30 18:03:09 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Mon, 30 Aug 2021 18:03:09 GMT Subject: RFR: JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v8] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has updated the pull request incrementally with one additional commit since the last revision: Skipping bytes instead of reading them Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5209/files - new: https://git.openjdk.java.net/jdk/pull/5209/files/b80bf8ea..98dc61e5 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=07 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5209&range=06-07 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/5209.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5209/head:pull/5209 PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Mon Aug 30 18:08:01 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Mon, 30 Aug 2021 18:08:01 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() [v7] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains two new commits since the last revision: - 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(WritableByteChannel) As suggested in 8265891 this change implements a performance optimization for ChannelInputStream.transferTo(): It uses FileChannel.transferTo(WritableByteChannel) in case the source channel actually is a FileChannel and the target channel is a WritableByteChannel, as in that case, the actual work of transferring bytes can potentially get offloaded to the operating system / file system. Signed-off-by: Markus Karg - Skipping bytes instead of reading them Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5179/files - new: https://git.openjdk.java.net/jdk/pull/5179/files/041e02fc..42387907 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=06 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=05-06 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/5179.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5179/head:pull/5179 PR: https://git.openjdk.java.net/jdk/pull/5179 From bpb at openjdk.java.net Mon Aug 30 23:59:30 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Mon, 30 Aug 2021 23:59:30 GMT Subject: RFR: JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v8] In-Reply-To: References: Message-ID: On Mon, 30 Aug 2021 18:03:09 GMT, Markus KARG wrote: >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and https://github.com/openjdk/jdk/pull/5179 and deliberately concentrates **only** on the case where both, source **and** target, are actually `FileChannel`s. Other types of channels will be discussed in future PRs. >> >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). > > Markus KARG has updated the pull request incrementally with one additional commit since the last revision: > > Skipping bytes instead of reading them > > Signed-off-by: Markus Karg As I understand it, if a `j.u.Random` instance is constructed from a given seed, then a series of invocations on that instance will always produce the same sequence of outputs. `RandomFactory` prints the seed in the test output and allows a seed to be provided to the test via a `-D` option. So if the test fails with a given combination of initial positions and buffer size, then it should be straightforward to reproduce the failure by providing the seed used when the test failed. ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From github.com+1701815+mkarg at openjdk.java.net Tue Aug 31 09:59:00 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Tue, 31 Aug 2021 09:59:00 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() [v8] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(WritableByteChannel) As suggested in 8265891 this change implements a performance optimization for ChannelInputStream.transferTo(): It uses FileChannel.transferTo(WritableByteChannel) in case the source channel actually is a FileChannel and the target channel is a WritableByteChannel, as in that case, the actual work of transferring bytes can potentially get offloaded to the operating system / file system. Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5179/files - new: https://git.openjdk.java.net/jdk/pull/5179/files/42387907..38875566 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=07 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=06-07 Stats: 12 lines in 1 file changed: 2 ins; 6 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/5179.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5179/head:pull/5179 PR: https://git.openjdk.java.net/jdk/pull/5179 From github.com+1701815+mkarg at openjdk.java.net Tue Aug 31 11:11:52 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Tue, 31 Aug 2021 11:11:52 GMT Subject: RFR: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo() [v9] In-Reply-To: References: Message-ID: > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. This PR is a spin-off from https://github.com/openjdk/jdk/pull/4263 and deliberately concentrates **only** on `FileChannel`s. Other types of channels will be discussed in future PRs. > > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. A rather similar approach in different use casse was recently implemented by https://github.com/openjdk/jdk/pull/5097 which was reported by even provide 5x performance (https://github.com/openjdk/jdk/pull/5097#issuecomment-897271997). Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: 8265891: ChannelInputStream.transferTo() uses FileChannel.transferTo(WritableByteChannel) As suggested in 8265891 this change implements a performance optimization for ChannelInputStream.transferTo(): It uses FileChannel.transferTo(WritableByteChannel) in case the source channel actually is a FileChannel and the target channel is a WritableByteChannel, as in that case, the actual work of transferring bytes can potentially get offloaded to the operating system / file system. Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/5179/files - new: https://git.openjdk.java.net/jdk/pull/5179/files/38875566..a507fd77 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=08 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5179&range=07-08 Stats: 5 lines in 1 file changed: 0 ins; 4 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/5179.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/5179/head:pull/5179 PR: https://git.openjdk.java.net/jdk/pull/5179 From github.com+1701815+mkarg at openjdk.java.net Tue Aug 31 13:54:25 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Tue, 31 Aug 2021 13:54:25 GMT Subject: RFR: JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) [v8] In-Reply-To: References: Message-ID: <-3Po7Bv73JJ3BLdjJBox-Chs64jBjHxQ5U4vqPvZlc4=.ba6cc277-edab-46b8-8851-c9ccf0124ca9@github.com> On Mon, 30 Aug 2021 23:56:45 GMT, Brian Burkhalter wrote: > As I understand it, if a `j.u.Random` instance is constructed from a given seed, then a series of invocations on that instance will always produce the same sequence of outputs. `RandomFactory` prints the seed in the test output and allows a seed to be provided to the test via a `-D` option. So if the test fails with a given combination of initial positions and buffer size, then it should be straightforward to reproduce the failure by providing the seed used when the test failed. Yes, given the fact it fails for ourselfs or at least for someone that knows that he has to tell us the printed seed. ------------- PR: https://git.openjdk.java.net/jdk/pull/5209 From mik3hall at gmail.com Tue Aug 31 15:24:26 2021 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 31 Aug 2021 10:24:26 -0500 Subject: TempFileHelper java.lang.IllegalArgumentException: Invalid prefix or suffix Message-ID: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> I am getting this error on code running on a recently downloaded 18 ea. The exception crawl java.lang.IllegalArgumentException: Invalid prefix or suffix at java.base/java.nio.file.TempFileHelper.generatePath(Unknown Source) at java.base/java.nio.file.TempFileHelper.create(Unknown Source) at java.base/java.nio.file.TempFileHelper.createTempFile(Unknown Source) at java.base/java.nio.file.Files.createTempFile(Unknown Source) at apple.applescript.AppleScriptEngine.eval(AppleScriptEngine.java:302) at apple.applescript.AppleScriptEngine.eval(AppleScriptEngine.java:267) at org.cmdline.cmds.Mps.main(Mps.java:28) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.cmdline.psuedoGestalt.Runner.invoke(Runner.java:209) at org.cmdline.psuedoGestalt.Runner.runStatic(Runner.java:236) at org.cmdline.psuedoGestalt.Runner.runMain(Runner.java:228) at org.cmdline.psuedoGestalt.Runner.run(Runner.java:146) I believe the error might be coming from TempFileHelper here? SecurityManager sm = System.getSecurityManager(); for (;;) { Path f; try { f = generatePath(prefix, suffix, dir); } catch (InvalidPathException e) { // don't reveal temporary directory location if (sm != null) throw new IllegalArgumentException("Invalid prefix or suffix"); throw e; Or possibly generatePath which I think has a similar error. The code causing the error is? tmpfile = Files.createTempFile("AppleScriptEngine.", ".scpt").toFile(); This hasn?t been changed by me. Doing this line jshell works. Running the code standalone outside my application works. I think the reason my application fails is that it includes a SecurityManager. First question. Is ?Invalid prefix or suffix? really the best error message for this? Second question; I have noticed things in the works for the elimination of SecurityManager. However, I thought this was still in the deprecation/warning stages. Should it be causing my code to fail if it is? From Alan.Bateman at oracle.com Tue Aug 31 15:52:10 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 31 Aug 2021 16:52:10 +0100 Subject: TempFileHelper java.lang.IllegalArgumentException: Invalid prefix or suffix In-Reply-To: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> References: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> Message-ID: On 31/08/2021 16:24, Michael Hall wrote: > I am getting this error on code running on a recently downloaded 18 ea. Do you see this with other JDK releases too? Is it possible that java.io.tmpdir has been set on the command line? > First question. Is ?Invalid prefix or suffix? really the best error message for this? A consequence of running with a security manager is that exceptions need to be careful to avoid leaking sensitive data, in this case the location of temporary files. So yes, this message is deliberate. -Alan From mik3hall at gmail.com Tue Aug 31 15:59:13 2021 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 31 Aug 2021 10:59:13 -0500 Subject: TempFileHelper java.lang.IllegalArgumentException: Invalid prefix or suffix In-Reply-To: References: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> Message-ID: <770EAD4B-78E5-4F05-A462-40075852265F@gmail.com> > On Aug 31, 2021, at 10:52 AM, Alan Bateman wrote: > > > > On 31/08/2021 16:24, Michael Hall wrote: >> I am getting this error on code running on a recently downloaded 18 ea. > > Do you see this with other JDK releases too? Is it possible that java.io.tmpdir has been set on the command line? I haven?t tried this yet. It means invoking jpackage indicating a earlier jdk. I can do this. Since I?ve sort of gathered 17 and 18 changes are sort of going in parallel right now I?ll try to go back to 16. > >> First question. Is ?Invalid prefix or suffix? really the best error message for this? > A consequence of running with a security manager is that exceptions need to be careful to avoid leaking sensitive data, in this case the location of temporary files. So yes, this message is deliberate. I think the actual nature of the error could be indicated without revealing the location of the files. The current jpackage invocation follows? ${PACKAGER} \ --verbose \ --input input \ --icon GenericApp.icns \ --resource-dir resources \ --name HalfPipe \ --main-jar halfpipe.jar \ --main-class us.hall.hp.common.LoaderLaunchStub \ --module-path mods \ --add-modules us.hall.eio,org.openjdk.nashorn,java.compiler,java.desktop,java.logging,java.management,java.prefs,java.se,java.rmi,java.scripting,java.sql,java.xml,jdk.attach,jdk.jshell,jdk.crypto.ec,jdk.jdeps,jdk.jcmd \ --java-options '-Xmx1024m -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider --add-exports org.openjdk.nashorn/org.openjdk.nashorn.tools=ALL-UNNAMED -Dapp.path=$APPDIR -Djava.security.policy=$APPDIR/all.policy -Dapple.laf.useScreenMenuBar=true -Dcom.apple.mrj.application.apple.menu.about.name=HalfPipe -Dapple.awt.application.name=HalfPipe -Dconsole=pane' \ --mac-package-identifier "us.hall.HalfPipe" \ --mac-sign \ --mac-signing-key-user-name "$SIGNING_CERT" \ --mac-entitlements "entitlements.xml" \ --jlink-options '--strip-debug --no-man-pages --no-header-files' From mik3hall at gmail.com Tue Aug 31 16:28:48 2021 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 31 Aug 2021 11:28:48 -0500 Subject: TempFileHelper java.lang.IllegalArgumentException: Invalid prefix or suffix In-Reply-To: <770EAD4B-78E5-4F05-A462-40075852265F@gmail.com> References: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> <770EAD4B-78E5-4F05-A462-40075852265F@gmail.com> Message-ID: <567C4033-4C6B-426B-A28A-AE45653A2213@gmail.com> > On Aug 31, 2021, at 10:59 AM, Michael Hall wrote: > >> Do you see this with other JDK releases too? Is it possible that java.io.tmpdir has been set on the command line? > > I haven?t tried this yet. It means invoking jpackage indicating a earlier jdk. I can do this. Since I?ve sort of gathered 17 and 18 changes are sort of going in parallel right now I?ll try to go back to 16. When you have some things working it?s not all that easy to fall back to where they were broken. But, yes, just changing to jdk16 and it works. No error. Changing to 17 at this point would be easy if you want me to try that? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mik3hall at gmail.com Tue Aug 31 17:27:55 2021 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 31 Aug 2021 12:27:55 -0500 Subject: TempFileHelper java.lang.IllegalArgumentException: Invalid prefix or suffix In-Reply-To: <567C4033-4C6B-426B-A28A-AE45653A2213@gmail.com> References: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> <770EAD4B-78E5-4F05-A462-40075852265F@gmail.com> <567C4033-4C6B-426B-A28A-AE45653A2213@gmail.com> Message-ID: <7CF3AE88-A40B-4ACD-A8D3-3F2591506FB6@gmail.com> > On Aug 31, 2021, at 11:28 AM, Michael Hall wrote: > > > >> On Aug 31, 2021, at 10:59 AM, Michael Hall > wrote: >> >>> Do you see this with other JDK releases too? Is it possible that java.io.tmpdir has been set on the command line? >> >> I haven?t tried this yet. It means invoking jpackage indicating a earlier jdk. I can do this. Since I?ve sort of gathered 17 and 18 changes are sort of going in parallel right now I?ll try to go back to 16. > > When you have some things working it?s not all that easy to fall back to where they were broken. > But, yes, just changing to jdk16 and it works. No error. > Changing to 17 at this point would be easy if you want me to try that? > Slightly different results? public class SecTemp { public static void main(String... args) throws java.io.IOException { System.setSecurityManager(new SecurityManager()); java.io.File tmpfile = java.nio.file.Files.createTempFile("AppleScriptEngine.", ".scpt").toFile(); } } java SecTemp WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by SecTemp (file:/Users/mjh/) WARNING: Please consider reporting this to the maintainers of SecTemp WARNING: System::setSecurityManager will be removed in a future release Exception in thread "main" java.lang.SecurityException: Unable to create temporary file or directory at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:141) at java.base/java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:159) at java.base/java.nio.file.Files.createTempFile(Files.java:923) at SecTemp.main(SecTemp.java:6) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Aug 31 17:39:25 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 31 Aug 2021 18:39:25 +0100 Subject: TempFileHelper java.lang.IllegalArgumentException: Invalid prefix or suffix In-Reply-To: <7CF3AE88-A40B-4ACD-A8D3-3F2591506FB6@gmail.com> References: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> <770EAD4B-78E5-4F05-A462-40075852265F@gmail.com> <567C4033-4C6B-426B-A28A-AE45653A2213@gmail.com> <7CF3AE88-A40B-4ACD-A8D3-3F2591506FB6@gmail.com> Message-ID: On 31/08/2021 18:27, Michael Hall wrote: > : > > Slightly different results? > > public class SecTemp { > > public static void main(String... args) throws java.io.IOException { > System.setSecurityManager(new SecurityManager()); > java.io.File tmpfile = > java.nio.file.Files.createTempFile("AppleScriptEngine.", > ".scpt").toFile(); > } > } > > java SecTemp > WARNING: A terminally deprecated method in java.lang.System has been > called > WARNING: System::setSecurityManager has been called by SecTemp > (file:/Users/mjh/) > WARNING: Please consider reporting this to the maintainers of SecTemp > WARNING: System::setSecurityManager will be removed in a future release > Exception in thread "main" java.lang.SecurityException: Unable to > create temporary file or directory > at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:141) > at > java.base/java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:159) > at java.base/java.nio.file.Files.createTempFile(Files.java:923) > at SecTemp.main(SecTemp.java:6) This is because it hasn't been granted permission to create files in ${java.io.tmpdir}. I would expect this behavior is every JDK release. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mik3hall at gmail.com Tue Aug 31 17:41:21 2021 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 31 Aug 2021 12:41:21 -0500 Subject: TempFileHelper java.lang.IllegalArgumentException: Invalid prefix or suffix In-Reply-To: References: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> <770EAD4B-78E5-4F05-A462-40075852265F@gmail.com> <567C4033-4C6B-426B-A28A-AE45653A2213@gmail.com> <7CF3AE88-A40B-4ACD-A8D3-3F2591506FB6@gmail.com> Message-ID: <3E2FC25A-196D-4695-9B3D-CDFEEAB1B775@gmail.com> > On Aug 31, 2021, at 12:39 PM, Alan Bateman wrote: > > On 31/08/2021 18:27, Michael Hall wrote: >> : >> >> Slightly different results? >> >> public class SecTemp { >> >> public static void main(String... args) throws java.io.IOException { >> System.setSecurityManager(new SecurityManager()); >> java.io.File tmpfile = java.nio.file.Files.createTempFile("AppleScriptEngine.", ".scpt").toFile(); >> } >> } >> >> java SecTemp >> WARNING: A terminally deprecated method in java.lang.System has been called >> WARNING: System::setSecurityManager has been called by SecTemp (file:/Users/mjh/ ) >> WARNING: Please consider reporting this to the maintainers of SecTemp >> WARNING: System::setSecurityManager will be removed in a future release >> Exception in thread "main" java.lang.SecurityException: Unable to create temporary file or directory >> at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:141) >> at java.base/java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:159) >> at java.base/java.nio.file.Files.createTempFile(Files.java:923) >> at SecTemp.main(SecTemp.java:6) > > This is because it hasn't been granted permission to create files in ${java.io.tmpdir}. I would expect this behavior is every JDK release. > > -Alan Ah, this one I didn?t test on any other release. If I provide an ?all? policy? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mik3hall at gmail.com Tue Aug 31 17:45:12 2021 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 31 Aug 2021 12:45:12 -0500 Subject: TempFileHelper java.lang.IllegalArgumentException: Invalid prefix or suffix In-Reply-To: <3E2FC25A-196D-4695-9B3D-CDFEEAB1B775@gmail.com> References: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> <770EAD4B-78E5-4F05-A462-40075852265F@gmail.com> <567C4033-4C6B-426B-A28A-AE45653A2213@gmail.com> <7CF3AE88-A40B-4ACD-A8D3-3F2591506FB6@gmail.com> <3E2FC25A-196D-4695-9B3D-CDFEEAB1B775@gmail.com> Message-ID: java -Djava.security.policy=HalfPipe/halfpipe_jpkg/outputdir/HalfPipe.app/Contents/app/all.policy SecTemp WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by SecTemp (file:/Users/mjh/) WARNING: Please consider reporting this to the maintainers of SecTemp WARNING: System::setSecurityManager will be removed in a future release No exception. So not a valid test case I guess. Different class loaders? From mik3hall at gmail.com Tue Aug 31 18:42:46 2021 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 31 Aug 2021 13:42:46 -0500 Subject: TempFileHelper java.lang.IllegalArgumentException: Invalid prefix or suffix In-Reply-To: References: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> <770EAD4B-78E5-4F05-A462-40075852265F@gmail.com> <567C4033-4C6B-426B-A28A-AE45653A2213@gmail.com> <7CF3AE88-A40B-4ACD-A8D3-3F2591506FB6@gmail.com> <3E2FC25A-196D-4695-9B3D-CDFEEAB1B775@gmail.com> Message-ID: > On Aug 31, 2021, at 12:45 PM, Michael Hall wrote: > > java -Djava.security.policy=HalfPipe/halfpipe_jpkg/outputdir/HalfPipe.app/Contents/app/all.policy SecTemp > WARNING: A terminally deprecated method in java.lang.System has been called > WARNING: System::setSecurityManager has been called by SecTemp (file:/Users/mjh/) > WARNING: Please consider reporting this to the maintainers of SecTemp > WARNING: System::setSecurityManager will be removed in a future release > > No exception. So not a valid test case I guess. Different class loaders? > This makes it valid. Probably mine have to look a little more? java -Djava.security.policy=HalfPipe/halfpipe_jpkg/outputdir/HalfPipe.app/Contents/app/all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider -cp .:HalfPipe/halfpipe_jpkg/outputdir/HalfPipe.app/Contents/app/macnio2.jar SecTemp WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by SecTemp (file:/Users/mjh/) WARNING: Please consider reporting this to the maintainers of SecTemp WARNING: System::setSecurityManager will be removed in a future release Exception in thread "main" java.lang.IllegalArgumentException: Invalid prefix or suffix at java.base/java.nio.file.TempFileHelper.generatePath(TempFileHelper.java:60) at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:125) at java.base/java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:159) at java.base/java.nio.file.Files.createTempFile(Files.java:923) at SecTemp.main(SecTemp.java:6) From mik3hall at gmail.com Tue Aug 31 19:42:53 2021 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 31 Aug 2021 14:42:53 -0500 Subject: TempFileHelper java.lang.IllegalArgumentException: Invalid prefix or suffix In-Reply-To: References: <4E9F4004-D3B8-4F4E-A715-2C99E1B63531@gmail.com> <770EAD4B-78E5-4F05-A462-40075852265F@gmail.com> <567C4033-4C6B-426B-A28A-AE45653A2213@gmail.com> <7CF3AE88-A40B-4ACD-A8D3-3F2591506FB6@gmail.com> <3E2FC25A-196D-4695-9B3D-CDFEEAB1B775@gmail.com> Message-ID: My custom default provider was trying to wrap a Path around a null returned platform provider getParent() instead of just returning null. > On Aug 31, 2021, at 1:42 PM, Michael Hall wrote: > > > >> On Aug 31, 2021, at 12:45 PM, Michael Hall wrote: >> >> java -Djava.security.policy=HalfPipe/halfpipe_jpkg/outputdir/HalfPipe.app/Contents/app/all.policy SecTemp >> WARNING: A terminally deprecated method in java.lang.System has been called >> WARNING: System::setSecurityManager has been called by SecTemp (file:/Users/mjh/) >> WARNING: Please consider reporting this to the maintainers of SecTemp >> WARNING: System::setSecurityManager will be removed in a future release >> >> No exception. So not a valid test case I guess. Different class loaders? >> > > This makes it valid. Probably mine have to look a little more? > > java -Djava.security.policy=HalfPipe/halfpipe_jpkg/outputdir/HalfPipe.app/Contents/app/all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider -cp .:HalfPipe/halfpipe_jpkg/outputdir/HalfPipe.app/Contents/app/macnio2.jar SecTemp > WARNING: A terminally deprecated method in java.lang.System has been called > WARNING: System::setSecurityManager has been called by SecTemp (file:/Users/mjh/) > WARNING: Please consider reporting this to the maintainers of SecTemp > WARNING: System::setSecurityManager will be removed in a future release > Exception in thread "main" java.lang.IllegalArgumentException: Invalid prefix or suffix > at java.base/java.nio.file.TempFileHelper.generatePath(TempFileHelper.java:60) > at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:125) > at java.base/java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:159) > at java.base/java.nio.file.Files.createTempFile(Files.java:923) > at SecTemp.main(SecTemp.java:6) > >