From alanb at openjdk.org Thu Sep 1 08:00:12 2022 From: alanb at openjdk.org (Alan Bateman) Date: Thu, 1 Sep 2022 08:00:12 GMT Subject: RFR: 8293121: (fs) Refactor UnixFileSystem copying into generic Unix, Linux, and BSD implementations [v3] In-Reply-To: References: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> Message-ID: On Wed, 31 Aug 2022 16:21:25 GMT, Brian Burkhalter wrote: >> Refactor `UnixFileSystem` and related classes to move implementations specific to Linux and BSD (macOS) into OS-specific classes and native code. Generic Unix implementations remain in existing `Unix*` files. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293121: Forgot to move UnixCopyFile.c to UnixFileSystem.c src/java.base/linux/native/libnio/fs/LinuxNativeDispatcher.c line 151: > 149: jint fd, jlong offset, jlong len, jint advice) > 150: { > 151: return posix_fadvise((int)fd, (off_t)offset, (off_t)len, (int)advice); I assume this should posix_fadvise64 as you'll need a off64_t on 32-bit systems. ------------- PR: https://git.openjdk.org/jdk/pull/10093 From alanb at openjdk.org Thu Sep 1 10:29:11 2022 From: alanb at openjdk.org (Alan Bateman) Date: Thu, 1 Sep 2022 10:29:11 GMT Subject: RFR: 8293121: (fs) Refactor UnixFileSystem copying into generic Unix, Linux, and BSD implementations [v3] In-Reply-To: References: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> Message-ID: On Wed, 31 Aug 2022 16:21:25 GMT, Brian Burkhalter wrote: >> Refactor `UnixFileSystem` and related classes to move implementations specific to Linux and BSD (macOS) into OS-specific classes and native code. Generic Unix implementations remain in existing `Unix*` files. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293121: Forgot to move UnixCopyFile.c to UnixFileSystem.c Overall I think the structure looks quite good. A lot of code has moved so it's hard to see what has changed so I focused on the copyFile and supporting methods as that seems to be where the only changes are. src/java.base/linux/classes/sun/nio/fs/LinuxFileSystem.java line 138: > 136: > 137: @Override > 138: protected void bufferedCopy(int dst, int src, long address, Everything is in sun.nio.fs and the existing methods aren't protected. I don't mind if we have the protected methods but it's a bit inconsistent to have it on some but not all of the methods that are intended to be used by subclasses. src/java.base/unix/classes/sun/nio/fs/UnixFileSystem.java line 428: > 426: > 427: // whether direct copying is supported on this platform > 428: private static volatile boolean directCopyNotSupported; Would it be possible to move MIN_BUFFER_SIZE and directCopyNotSupported to the top so they are declared before the instance fields? test/jdk/java/nio/file/Files/CopyAndMove.java line 25: > 23: > 24: /* @test > 25: * @bug 4313887 6838333 6917021 7006126 6950237 8006645 8201407 8293121 8267820 Did you mean to include this test update in this PR? ------------- PR: https://git.openjdk.org/jdk/pull/10093 From bpb at openjdk.org Thu Sep 1 14:53:29 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 1 Sep 2022 14:53:29 GMT Subject: RFR: 8293121: (fs) Refactor UnixFileSystem copying into generic Unix, Linux, and BSD implementations [v3] In-Reply-To: References: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> Message-ID: <87s7g43gDanOEy0g3r3egSPHqTvKjsXnwyW5Gtvo1FA=.03f358b1-8569-4d4f-ba74-2d972a1db956@github.com> On Thu, 1 Sep 2022 10:25:29 GMT, Alan Bateman wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8293121: Forgot to move UnixCopyFile.c to UnixFileSystem.c > > src/java.base/linux/classes/sun/nio/fs/LinuxFileSystem.java line 138: > >> 136: >> 137: @Override >> 138: protected void bufferedCopy(int dst, int src, long address, > > Everything is in sun.nio.fs and the existing methods aren't protected. I don't mind if we have the protected methods but it's a bit inconsistent to have it on some but not all of the methods that are intended to be used by subclasses. I was thinking that `protected` would suggest that the method should likely be overridden. > test/jdk/java/nio/file/Files/CopyAndMove.java line 25: > >> 23: >> 24: /* @test >> 25: * @bug 4313887 6838333 6917021 7006126 6950237 8006645 8201407 8293121 8267820 > > Did you mean to include this test update in this PR? I was thinking of the coverage but as this really adds a CoW file system if available then I think it better to leave it in the other PR about cloning on Linux. ------------- PR: https://git.openjdk.org/jdk/pull/10093 From bpb at openjdk.org Thu Sep 1 16:16:34 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 1 Sep 2022 16:16:34 GMT Subject: RFR: 8293121: (fs) Refactor UnixFileSystem copying into generic Unix, Linux, and BSD implementations [v4] In-Reply-To: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> References: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> Message-ID: > Refactor `UnixFileSystem` and related classes to move implementations specific to Linux and BSD (macOS) into OS-specific classes and native code. Generic Unix implementations remain in existing `Unix*` files. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8293121: Address review comments ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10093/files - new: https://git.openjdk.org/jdk/pull/10093/files/be36c5a6..b6cb9490 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10093&range=03 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10093&range=02-03 Stats: 101 lines in 5 files changed: 7 ins; 76 del; 18 mod Patch: https://git.openjdk.org/jdk/pull/10093.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10093/head:pull/10093 PR: https://git.openjdk.org/jdk/pull/10093 From bpb at openjdk.org Thu Sep 1 16:16:35 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 1 Sep 2022 16:16:35 GMT Subject: RFR: 8293121: (fs) Refactor UnixFileSystem copying into generic Unix, Linux, and BSD implementations [v3] In-Reply-To: <87s7g43gDanOEy0g3r3egSPHqTvKjsXnwyW5Gtvo1FA=.03f358b1-8569-4d4f-ba74-2d972a1db956@github.com> References: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> <87s7g43gDanOEy0g3r3egSPHqTvKjsXnwyW5Gtvo1FA=.03f358b1-8569-4d4f-ba74-2d972a1db956@github.com> Message-ID: On Thu, 1 Sep 2022 14:50:59 GMT, Brian Burkhalter wrote: >> src/java.base/linux/classes/sun/nio/fs/LinuxFileSystem.java line 138: >> >>> 136: >>> 137: @Override >>> 138: protected void bufferedCopy(int dst, int src, long address, >> >> Everything is in sun.nio.fs and the existing methods aren't protected. I don't mind if we have the protected methods but it's a bit inconsistent to have it on some but not all of the methods that are intended to be used by subclasses. > > I was thinking that `protected` would suggest that the method should likely be overridden. Removed `protected` in b6cb94901d1bbaa34917a5aaf381ecf700282c54. >> test/jdk/java/nio/file/Files/CopyAndMove.java line 25: >> >>> 23: >>> 24: /* @test >>> 25: * @bug 4313887 6838333 6917021 7006126 6950237 8006645 8201407 8293121 8267820 >> >> Did you mean to include this test update in this PR? > > I was thinking of the coverage but as this really adds a CoW file system if available then I think it better to leave it in the other PR about cloning on Linux. Removed in b6cb94901d1bbaa34917a5aaf381ecf700282c54. ------------- PR: https://git.openjdk.org/jdk/pull/10093 From bpb at openjdk.org Thu Sep 1 16:16:36 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 1 Sep 2022 16:16:36 GMT Subject: RFR: 8293121: (fs) Refactor UnixFileSystem copying into generic Unix, Linux, and BSD implementations [v3] In-Reply-To: References: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> Message-ID: On Thu, 1 Sep 2022 07:56:32 GMT, Alan Bateman wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8293121: Forgot to move UnixCopyFile.c to UnixFileSystem.c > > src/java.base/linux/native/libnio/fs/LinuxNativeDispatcher.c line 151: > >> 149: jint fd, jlong offset, jlong len, jint advice) >> 150: { >> 151: return posix_fadvise((int)fd, (off_t)offset, (off_t)len, (int)advice); > > I assume this should be posix_fadvise64 as you'll need a off64_t on 32-bit systems. Fixed in b6cb94901d1bbaa34917a5aaf381ecf700282c54. > src/java.base/unix/classes/sun/nio/fs/UnixFileSystem.java line 428: > >> 426: >> 427: // whether direct copying is supported on this platform >> 428: private static volatile boolean directCopyNotSupported; > > Would it be possible to move MIN_BUFFER_SIZE and directCopyNotSupported to the top so they are declared before the instance fields? Fixed in b6cb94901d1bbaa34917a5aaf381ecf700282c54. ------------- PR: https://git.openjdk.org/jdk/pull/10093 From alanb at openjdk.org Thu Sep 1 18:34:24 2022 From: alanb at openjdk.org (Alan Bateman) Date: Thu, 1 Sep 2022 18:34:24 GMT Subject: RFR: 8293121: (fs) Refactor UnixFileSystem copying into generic Unix, Linux, and BSD implementations [v4] In-Reply-To: References: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> Message-ID: <69drPL8hkxfeUavgTcsefNy0WTUvMEL_Ui4a_EKLtJw=.ad7e8f79-12e4-420c-a271-ac3ff2e70579@github.com> On Thu, 1 Sep 2022 16:16:34 GMT, Brian Burkhalter wrote: >> Refactor `UnixFileSystem` and related classes to move implementations specific to Linux and BSD (macOS) into OS-specific classes and native code. Generic Unix implementations remain in existing `Unix*` files. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293121: Address review comments Thanks for the updates, I don't have any comments/issues. ------------- Marked as reviewed by alanb (Reviewer). PR: https://git.openjdk.org/jdk/pull/10093 From bpb at openjdk.org Thu Sep 1 18:34:25 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 1 Sep 2022 18:34:25 GMT Subject: RFR: 8293121: (fs) Refactor UnixFileSystem copying into generic Unix, Linux, and BSD implementations [v4] In-Reply-To: <69drPL8hkxfeUavgTcsefNy0WTUvMEL_Ui4a_EKLtJw=.ad7e8f79-12e4-420c-a271-ac3ff2e70579@github.com> References: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> <69drPL8hkxfeUavgTcsefNy0WTUvMEL_Ui4a_EKLtJw=.ad7e8f79-12e4-420c-a271-ac3ff2e70579@github.com> Message-ID: On Thu, 1 Sep 2022 18:31:17 GMT, Alan Bateman wrote: > Thanks for the updates, I don't have any comments/issues. Thanks for the review! ------------- PR: https://git.openjdk.org/jdk/pull/10093 From bpb at openjdk.org Thu Sep 1 22:23:49 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 1 Sep 2022 22:23:49 GMT Subject: Integrated: 8293121: (fs) Refactor UnixFileSystem copying into generic Unix, Linux, and BSD implementations In-Reply-To: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> References: <4sMmGghqBMpTXTI5cK9rCKG1Rdt8IDdumWN0_x4QeYU=.27c2230d-8775-4a5a-b04e-c1fcad494841@github.com> Message-ID: On Wed, 31 Aug 2022 02:28:44 GMT, Brian Burkhalter wrote: > Refactor `UnixFileSystem` and related classes to move implementations specific to Linux and BSD (macOS) into OS-specific classes and native code. Generic Unix implementations remain in existing `Unix*` files. This pull request has now been integrated. Changeset: 0a4d0cee Author: Brian Burkhalter URL: https://git.openjdk.org/jdk/commit/0a4d0cee9ffd77eaa26493f20bac4bfaccd48c3b Stats: 2088 lines in 12 files changed: 1134 ins; 937 del; 17 mod 8293121: (fs) Refactor UnixFileSystem copying into generic Unix, Linux, and BSD implementations Reviewed-by: alanb ------------- PR: https://git.openjdk.org/jdk/pull/10093 From bpb at openjdk.org Fri Sep 2 00:26:31 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 2 Sep 2022 00:26:31 GMT Subject: RFR: 8264744: (fs) Use file cloning in Linux version of Files::copy method [v10] In-Reply-To: References: Message-ID: > Add file cloning to `java.nio.channels.FileChannel::transferTo` and `java.nio.file.Files.copy(Path,Path)`. Brian Burkhalter has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 14 commits: - Merge - 8264744: Refactor UnixCopyFile::copyFile into UnixFileSystem; refactor OS-specific code to {Bsd,Linux}FileSystem - Merge - 8264744: Add mode to clone() and other requested changes - 8264744: Refactor {Bsd,Linux}CopyFile::clone into {Bsd,Linux}FileSystem::clone - Merge - 8264744: Remove CloneFile classes; refactor to use {Bsd,Linux}CopyFile - Merge - 8264744: Fix typo desintation -> destination - 8264744: Refactor into UnixCopyFile hierarchy - ... and 4 more: https://git.openjdk.org/jdk/compare/0a4d0cee...4260d000 ------------- Changes: https://git.openjdk.org/jdk/pull/9486/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=9486&range=09 Stats: 243 lines in 9 files changed: 223 ins; 14 del; 6 mod Patch: https://git.openjdk.org/jdk/pull/9486.diff Fetch: git fetch https://git.openjdk.org/jdk pull/9486/head:pull/9486 PR: https://git.openjdk.org/jdk/pull/9486 From bpb at openjdk.org Fri Sep 2 01:21:13 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 2 Sep 2022 01:21:13 GMT Subject: RFR: 8264744: (fs) Use file cloning in Linux version of Files::copy method [v11] In-Reply-To: References: Message-ID: > Add file cloning to `java.nio.channels.FileChannel::transferTo` and `java.nio.file.Files.copy(Path,Path)`. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8264744: Reinstate copy_file_range in Linux; remove vestiges of macOS clonefile ------------- Changes: - all: https://git.openjdk.org/jdk/pull/9486/files - new: https://git.openjdk.org/jdk/pull/9486/files/4260d000..2a98f0fe Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=9486&range=10 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=9486&range=09-10 Stats: 192 lines in 8 files changed: 46 ins; 145 del; 1 mod Patch: https://git.openjdk.org/jdk/pull/9486.diff Fetch: git fetch https://git.openjdk.org/jdk pull/9486/head:pull/9486 PR: https://git.openjdk.org/jdk/pull/9486 From bpb at openjdk.org Fri Sep 2 01:21:16 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 2 Sep 2022 01:21:16 GMT Subject: RFR: 8264744: (fs) Use file cloning in Linux version of Files::copy method [v9] In-Reply-To: References: Message-ID: On Tue, 30 Aug 2022 16:36:00 GMT, Brian Burkhalter wrote: >> Brian Burkhalter has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 13 commits: >> >> - 8264744: Refactor UnixCopyFile::copyFile into UnixFileSystem; refactor OS-specific code to {Bsd,Linux}FileSystem >> - Merge >> - 8264744: Add mode to clone() and other requested changes >> - 8264744: Refactor {Bsd,Linux}CopyFile::clone into {Bsd,Linux}FileSystem::clone >> - Merge >> - 8264744: Remove CloneFile classes; refactor to use {Bsd,Linux}CopyFile >> - Merge >> - 8264744: Fix typo desintation -> destination >> - 8264744: Refactor into UnixCopyFile hierarchy >> - Merge >> - ... and 3 more: https://git.openjdk.org/jdk/compare/c74b6d45...ecce2093 > > src/java.base/linux/native/libnio/fs/LinuxFileSystem.c line 96: > >> 94: case EINVAL: >> 95: case EXDEV: >> 96: // ignore and try sendfile() > > `ENOSYS` will need to be handled as done [here](https://github.com/openjdk/jdk/pull/10055/files#diff-4fdc393c9569db35d35d0b5698849aa939ba395ec5a685befadbb2bacbaddb3e). Addressed in 2a98f0fe1acfbcf36151d9523ecf097c41100b78. ------------- PR: https://git.openjdk.org/jdk/pull/9486 From bpb at openjdk.org Fri Sep 2 01:37:59 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 2 Sep 2022 01:37:59 GMT Subject: RFR: 8264744: (fs) Use file cloning in Linux version of Files::copy method [v12] In-Reply-To: References: Message-ID: <4TuOsG7Nv4mjnYMC2hPUFM_kZi9XRHxFIQouyjv-vpQ=.5c5957cb-84db-4e55-ad3e-b35f2a62bd35@github.com> > Add file cloning to `java.nio.channels.FileChannel::transferTo` and `java.nio.file.Files.copy(Path,Path)`. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8264744: Reduce noise of negligible diffs in UnixFileSystem ------------- Changes: - all: https://git.openjdk.org/jdk/pull/9486/files - new: https://git.openjdk.org/jdk/pull/9486/files/2a98f0fe..62461a2d Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=9486&range=11 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=9486&range=10-11 Stats: 2 lines in 2 files changed: 1 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jdk/pull/9486.diff Fetch: git fetch https://git.openjdk.org/jdk pull/9486/head:pull/9486 PR: https://git.openjdk.org/jdk/pull/9486 From mkartashev at openjdk.org Fri Sep 2 08:02:37 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Fri, 2 Sep 2022 08:02:37 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) Message-ID: This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. Some things of note: * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. ### Testing * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). * Also verified that new tests pass on Linux and Windows. * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. ------------- Commit messages: - 8293067: (fs) Implement WatchService using system library (macOS) Changes: https://git.openjdk.org/jdk/pull/10140/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8293067 Stats: 1338 lines in 10 files changed: 1329 ins; 3 del; 6 mod Patch: https://git.openjdk.org/jdk/pull/10140.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10140/head:pull/10140 PR: https://git.openjdk.org/jdk/pull/10140 From alanb at openjdk.org Fri Sep 2 08:15:41 2022 From: alanb at openjdk.org (Alan Bateman) Date: Fri, 2 Sep 2022 08:15:41 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: References: Message-ID: On Fri, 2 Sep 2022 07:54:32 GMT, Maxim Kartashev wrote: > This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. > > Some things of note: > * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. > * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). > * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). > * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). > * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. > > This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. > > ### Testing > > * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). > * Also verified that new tests pass on Linux and Windows. > * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. Thanks for this PR, I plan to review this but it will take some time. Just to set expectations that the patch will likely through through several iterations as there will be many issues to discuss. > This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. I think it is still used on on AIX but there is someone from IBM working on that. It may be that we will have to keep it for new ports. For macOS then we might have to introduce a system property to use it, at least until there is confidence with the new implementation. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From mkartashev at openjdk.org Fri Sep 2 09:30:44 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Fri, 2 Sep 2022 09:30:44 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: References: Message-ID: On Fri, 2 Sep 2022 08:13:35 GMT, Alan Bateman wrote: > For macOS then we might have to introduce a system property to use it, at least until there is confidence with the new implementation. I actually had that property, but pulled it out before submitting the pull request, so it's going to be very easy to re-introduce it. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From rriggs at openjdk.org Fri Sep 2 13:58:56 2022 From: rriggs at openjdk.org (Roger Riggs) Date: Fri, 2 Sep 2022 13:58:56 GMT Subject: RFR: 8057113: (fs) Path should have a method to obtain the filename extension [v18] In-Reply-To: References: Message-ID: On Mon, 1 Aug 2022 21:54:22 GMT, Brian Burkhalter wrote: >> Resurrection of the proposal to add a method to obtain the filename extension originated in PR [2319](https://github.com/openjdk/jdk/pull/2319). > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8057113: Remove API note Lets keep this alive and provide and resolve remaining issues. ------------- PR: https://git.openjdk.org/jdk/pull/8066 From bpb at openjdk.org Sat Sep 3 01:06:43 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Sat, 3 Sep 2022 01:06:43 GMT Subject: RFR: 8293331: Refactor FileChannelImpl into operating system-specific components Message-ID: Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. ------------- Commit messages: - 8293331: Refactor FileChannelImpl into operating system-specific components Changes: https://git.openjdk.org/jdk/pull/10154/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8293331 Stats: 1294 lines in 13 files changed: 865 ins; 391 del; 38 mod Patch: https://git.openjdk.org/jdk/pull/10154.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10154/head:pull/10154 PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Sat Sep 3 01:06:44 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Sat, 3 Sep 2022 01:06:44 GMT Subject: RFR: 8293331: Refactor FileChannelImpl into operating system-specific components In-Reply-To: References: Message-ID: On Sat, 3 Sep 2022 00:59:07 GMT, Brian Burkhalter wrote: > Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. The initial implementation moves most of `FileChannelImpl` to `sun.nio.ch.AbstractFileChannelImpl` and requires separate but simple `FileChannelImpl.java` and `FileChannelImpl.c` files for each operating system. The AIX, Linux, and macOS versions extend an intermediate class `sun.nio.ch.UnixFileChannelImpl` which itself extends `AbstractFileChannelImpl`. All tests in the `jdk_nio` test group pass on Linux, macOS, and Windows. Although an AIX implementation is provided, it will have to be verified at a later date by someone with access to the requisite platform. ------------- PR: https://git.openjdk.org/jdk/pull/10154 From alanb at openjdk.org Sat Sep 3 06:17:41 2022 From: alanb at openjdk.org (Alan Bateman) Date: Sat, 3 Sep 2022 06:17:41 GMT Subject: RFR: 8293331: Refactor FileChannelImpl into operating system-specific components In-Reply-To: References: Message-ID: On Sat, 3 Sep 2022 00:59:07 GMT, Brian Burkhalter wrote: > Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. FileChannel delegates to a FileDispatcher so there is already a place for OS specific code (both java and native code). I think we should try to lean on that before introducing a parallel hierarchy of classes. ------------- PR: https://git.openjdk.org/jdk/pull/10154 From alanb at openjdk.org Sat Sep 3 13:51:41 2022 From: alanb at openjdk.org (Alan Bateman) Date: Sat, 3 Sep 2022 13:51:41 GMT Subject: RFR: 8264744: (fs) Use file cloning in Linux version of Files::copy method [v12] In-Reply-To: <4TuOsG7Nv4mjnYMC2hPUFM_kZi9XRHxFIQouyjv-vpQ=.5c5957cb-84db-4e55-ad3e-b35f2a62bd35@github.com> References: <4TuOsG7Nv4mjnYMC2hPUFM_kZi9XRHxFIQouyjv-vpQ=.5c5957cb-84db-4e55-ad3e-b35f2a62bd35@github.com> Message-ID: On Fri, 2 Sep 2022 01:37:59 GMT, Brian Burkhalter wrote: >> Add file cloning to `java.nio.channels.FileChannel::transferTo` and `java.nio.file.Files.copy(Path,Path)`. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8264744: Reduce noise of negligible diffs in UnixFileSystem The updated implementation changes in 62461a2d look okay but I'm not sure about the updates to the CopyAndMove test. If I read it correctly, it tries to find the mount point where a COW file system is mounted and use it as an additional test directory. I don't think we should want tests trying to use these locations as test directories, even if running with permissions. ------------- PR: https://git.openjdk.org/jdk/pull/9486 From duke at openjdk.org Sun Sep 4 21:02:44 2022 From: duke at openjdk.org (duke) Date: Sun, 4 Sep 2022 21:02:44 GMT Subject: Withdrawn: 8278268 - (ch) InputStream returned by Channels.newInputStream should have fast path for FileChannel targets In-Reply-To: References: Message-ID: On Sun, 5 Dec 2021 13:04:22 GMT, Markus KARG wrote: > This sub-issue defines the work to be done to implement JDK-8265891 solely for the particular case of FileChannel.transferFrom(ReadableByteChannel), including special treatment of SelectableByteChannel. This pull request has been closed without being integrated. ------------- PR: https://git.openjdk.org/jdk/pull/6711 From bpb at openjdk.org Tue Sep 6 17:20:59 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 6 Sep 2022 17:20:59 GMT Subject: RFR: 8264744: (fs) Use file cloning in Linux version of Files::copy method [v13] In-Reply-To: References: Message-ID: > Add file cloning to `java.nio.channels.FileChannel::transferTo` and `java.nio.file.Files.copy(Path,Path)`. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8264744: Remove search for CoW volumes; run same directory tests on test.dir if it has a different format ------------- Changes: - all: https://git.openjdk.org/jdk/pull/9486/files - new: https://git.openjdk.org/jdk/pull/9486/files/62461a2d..f1da9461 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=9486&range=12 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=9486&range=11-12 Stats: 55 lines in 1 file changed: 1 ins; 52 del; 2 mod Patch: https://git.openjdk.org/jdk/pull/9486.diff Fetch: git fetch https://git.openjdk.org/jdk pull/9486/head:pull/9486 PR: https://git.openjdk.org/jdk/pull/9486 From alanb at openjdk.org Tue Sep 6 17:26:48 2022 From: alanb at openjdk.org (Alan Bateman) Date: Tue, 6 Sep 2022 17:26:48 GMT Subject: RFR: 8264744: (fs) Use file cloning in Linux version of Files::copy method [v13] In-Reply-To: References: Message-ID: On Tue, 6 Sep 2022 17:20:59 GMT, Brian Burkhalter wrote: >> Add file cloning to `java.nio.channels.FileChannel::transferTo` and `java.nio.file.Files.copy(Path,Path)`. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8264744: Remove search for CoW volumes; run same directory tests on test.dir if it has a different format Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.org/jdk/pull/9486 From bpb at openjdk.org Tue Sep 6 18:04:43 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 6 Sep 2022 18:04:43 GMT Subject: RFR: 8293331: Refactor FileChannelImpl into operating system-specific components In-Reply-To: References: Message-ID: On Sat, 3 Sep 2022 06:13:49 GMT, Alan Bateman wrote: > FileChannel delegates to a FileDispatcher so there is already a place for OS specific code (both java and native code). I think we should try to lean on that before introducing a parallel hierarchy of classes. Having thought more about it over the long weekend I don't like this proposal as it stands either. The native methods as-is however I thought are too long for the usual `FileDispatcher` standard. I will however investigate this alternative. ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Tue Sep 6 18:14:11 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 6 Sep 2022 18:14:11 GMT Subject: Integrated: 8264744: (fs) Use file cloning in Linux version of Files::copy method In-Reply-To: References: Message-ID: On Wed, 13 Jul 2022 23:25:00 GMT, Brian Burkhalter wrote: > Add file cloning to `java.nio.channels.FileChannel::transferTo` and `java.nio.file.Files.copy(Path,Path)`. This pull request has now been integrated. Changeset: da596182 Author: Brian Burkhalter URL: https://git.openjdk.org/jdk/commit/da596182a494a36d37030f18328e52e525fc3565 Stats: 89 lines in 4 files changed: 73 ins; 13 del; 3 mod 8264744: (fs) Use file cloning in Linux version of Files::copy method Reviewed-by: alanb ------------- PR: https://git.openjdk.org/jdk/pull/9486 From duke at openjdk.org Tue Sep 6 20:04:49 2022 From: duke at openjdk.org (Mike Hearn) Date: Tue, 6 Sep 2022 20:04:49 GMT Subject: RFR: 8264744: (fs) Use file cloning in Linux version of Files::copy method [v13] In-Reply-To: References: Message-ID: On Tue, 6 Sep 2022 17:20:59 GMT, Brian Burkhalter wrote: >> Add file cloning to `java.nio.channels.FileChannel::transferTo` and `java.nio.file.Files.copy(Path,Path)`. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8264744: Remove search for CoW volumes; run same directory tests on test.dir if it has a different format Thank you! I'm the one who filed the original webbug report (some day I must figure out how to get direct access to the JDK bug tracker). This wasn't a theoretical request, it was motivated by a real product that my company has just launched called [Conveyor](https://conveyor.hydraulic.dev/). Conveyor is a sort of jpackage++ and is structured as a parallel build system in which intermediate steps are cached. The different tasks sometimes have to copy large files around (like JDKs, JARs etc) and using kernel level COW cloning will yield a large boost to performance on filesystems that support it. ------------- PR: https://git.openjdk.org/jdk/pull/9486 From bpb at openjdk.org Tue Sep 6 22:53:43 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 6 Sep 2022 22:53:43 GMT Subject: RFR: 8293331: Refactor FileChannelImpl into operating system-specific components [v2] In-Reply-To: References: Message-ID: > Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. Brian Burkhalter has updated the pull request incrementally with two additional commits since the last revision: - 8293331: Move OS-specific code to FileDispatcher; remove OS-specific FileChannelImpl classes - 8293331: correct an error in the AIX FileChannelImpl.c ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10154/files - new: https://git.openjdk.org/jdk/pull/10154/files/1d9b50c7..4a3e5fc7 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=00-01 Stats: 1682 lines in 17 files changed: 581 ins; 1062 del; 39 mod Patch: https://git.openjdk.org/jdk/pull/10154.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10154/head:pull/10154 PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Wed Sep 7 00:55:39 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 7 Sep 2022 00:55:39 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method Message-ID: Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. ------------- Commit messages: - 8293122: (fs) Use file cloning in macOS version of Files::copy method Changes: https://git.openjdk.org/jdk/pull/10188/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10188&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8293122 Stats: 231 lines in 7 files changed: 229 ins; 0 del; 2 mod Patch: https://git.openjdk.org/jdk/pull/10188.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10188/head:pull/10188 PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Wed Sep 7 00:55:40 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 7 Sep 2022 00:55:40 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method In-Reply-To: References: Message-ID: On Wed, 7 Sep 2022 00:48:31 GMT, Brian Burkhalter wrote: > Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 110: > 108: switch (x.errno()) { > 109: case ENOTSUP: // cloning not supported by filesystem > 110: return IOStatus.UNSUPPORTED; It might be that `ENOTSUP` should instead result in `UNSUPPORTED_CASE` and that the static variable `cloneFileNotSupported` is unnecessary. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From alanb at openjdk.org Wed Sep 7 15:17:49 2022 From: alanb at openjdk.org (Alan Bateman) Date: Wed, 7 Sep 2022 15:17:49 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method In-Reply-To: References: Message-ID: On Wed, 7 Sep 2022 00:48:31 GMT, Brian Burkhalter wrote: > Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 101: > 99: BsdFileStore bfs = (BsdFileStore)provider().getFileStore(src); > 100: if (!bfs.equals(provider().getFileStore(dst.getParent())) || > 101: !bfs.supportsCloning()) Is 'dst' a relative path, in which case you can't rely on getParent. In any case, I don't think we should be checking the FileStore here as that means finding the mount entry. If we do clonefile then I think you'll have to rely on the EXDEV error. src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 141: > 139: // cancellation is not possible, or attributes are not to be copied > 140: if (!cloneFileNotSupported && addressToPollForCancel == 0 && > 141: flags.copyPosixAttributes) { This will need some thought as this means it will only be used when Files.copy is invoked with the COPY_ATTRIBUTES option. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Wed Sep 7 17:02:45 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 7 Sep 2022 17:02:45 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method In-Reply-To: References: Message-ID: <6-x5WK0eFs1tAM4xeDxE-4FLhalshmUbZq7V70f6GKU=.6cb07e7d-6264-4086-b455-f9563dba0048@github.com> On Wed, 7 Sep 2022 15:14:18 GMT, Alan Bateman wrote: >> Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. > > src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 141: > >> 139: // cancellation is not possible, or attributes are not to be copied >> 140: if (!cloneFileNotSupported && addressToPollForCancel == 0 && >> 141: flags.copyPosixAttributes) { > > This will need some thought as this means it will only be used when Files.copy is invoked with the COPY_ATTRIBUTES option. `clonefile` appears to copy all attributes so if it is acceptable for that to occur if `COPY_ATTRIBUTES` is not set, then the `flags.copyPosixAttributes` check could be removed. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Wed Sep 7 17:47:14 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 7 Sep 2022 17:47:14 GMT Subject: RFR: 8293456: runtime/os/TestTracePageSizes.java sub-tests fail with "AssertionError: No memory range found for address: NNNN" Message-ID: Do not use direct copy if the source size is zero as this may fail on some configurations but appear to succeed. ------------- Commit messages: - 8293456: runtime/os/TestTracePageSizes.java sub-tests fail with "AssertionError: No memory range found for address: NNNN" Changes: https://git.openjdk.org/jdk/pull/10204/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10204&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8293456 Stats: 10 lines in 2 files changed: 3 ins; 0 del; 7 mod Patch: https://git.openjdk.org/jdk/pull/10204.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10204/head:pull/10204 PR: https://git.openjdk.org/jdk/pull/10204 From bpb at openjdk.org Wed Sep 7 17:53:02 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 7 Sep 2022 17:53:02 GMT Subject: RFR: 8293456: runtime/os/TestTracePageSizes.java sub-tests fail with "AssertionError: No memory range found for address: NNNN" [v2] In-Reply-To: References: Message-ID: > Do not use direct copy if the source size is zero as this may fail on some configurations but appear to succeed. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8293456: Fix typo ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10204/files - new: https://git.openjdk.org/jdk/pull/10204/files/036408e4..6e5c69f7 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10204&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10204&range=00-01 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jdk/pull/10204.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10204/head:pull/10204 PR: https://git.openjdk.org/jdk/pull/10204 From alanb at openjdk.org Wed Sep 7 17:53:02 2022 From: alanb at openjdk.org (Alan Bateman) Date: Wed, 7 Sep 2022 17:53:02 GMT Subject: RFR: 8293456: runtime/os/TestTracePageSizes.java sub-tests fail with "AssertionError: No memory range found for address: NNNN" [v2] In-Reply-To: References: Message-ID: On Wed, 7 Sep 2022 17:49:49 GMT, Brian Burkhalter wrote: >> Do not use direct copy if the source size is zero as this may fail on some configurations but appear to succeed. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293456: Fix typo Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.org/jdk/pull/10204 From lancea at openjdk.org Wed Sep 7 18:11:43 2022 From: lancea at openjdk.org (Lance Andersen) Date: Wed, 7 Sep 2022 18:11:43 GMT Subject: RFR: 8293456: runtime/os/TestTracePageSizes.java sub-tests fail with "AssertionError: No memory range found for address: NNNN" [v2] In-Reply-To: References: Message-ID: On Wed, 7 Sep 2022 17:53:02 GMT, Brian Burkhalter wrote: >> Do not use direct copy if the source size is zero as this may fail on some configurations but appear to succeed. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293456: Fix typo Marked as reviewed by lancea (Reviewer). ------------- PR: https://git.openjdk.org/jdk/pull/10204 From bpb at openjdk.org Wed Sep 7 18:57:50 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 7 Sep 2022 18:57:50 GMT Subject: Integrated: 8293456: runtime/os/TestTracePageSizes.java sub-tests fail with "AssertionError: No memory range found for address: NNNN" In-Reply-To: References: Message-ID: On Wed, 7 Sep 2022 17:41:36 GMT, Brian Burkhalter wrote: > Do not use direct copy if the source size is zero as this may fail on some configurations but appear to succeed. This pull request has now been integrated. Changeset: 76df73b3 Author: Brian Burkhalter URL: https://git.openjdk.org/jdk/commit/76df73b390d0db7779935dd254f6378b6884a79d Stats: 10 lines in 2 files changed: 3 ins; 0 del; 7 mod 8293456: runtime/os/TestTracePageSizes.java sub-tests fail with "AssertionError: No memory range found for address: NNNN" Reviewed-by: alanb, lancea ------------- PR: https://git.openjdk.org/jdk/pull/10204 From bpb at openjdk.org Wed Sep 7 21:54:44 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 7 Sep 2022 21:54:44 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: References: Message-ID: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> On Fri, 2 Sep 2022 07:54:32 GMT, Maxim Kartashev wrote: > This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. > > Some things of note: > * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. > * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). > * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). > * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). > * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. > > This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. > > ### Testing > > * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). > * Also verified that new tests pass on Linux and Windows. > * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. In our CI system `test/jdk/java/nio/file/WatchService/Move.java` appears to time out consistently, although I have not seen it happen on my local machine. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From mkartashev at openjdk.org Thu Sep 8 07:24:44 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Thu, 8 Sep 2022 07:24:44 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> References: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> Message-ID: On Wed, 7 Sep 2022 21:51:02 GMT, Brian Burkhalter wrote: > In our CI system `test/jdk/java/nio/file/WatchService/Move.java` appears to time out consistently, although I have not seen it happen on my local machine. Can you provide the `.jtr` file, please? I haven't seen the test hang, but by inserting different delays different systems can expose synchronization issues, of course. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From mik3hall at gmail.com Thu Sep 8 13:33:26 2022 From: mik3hall at gmail.com (Michael Hall) Date: Thu, 8 Sep 2022 08:33:26 -0500 Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> References: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> Message-ID: > On Sep 7, 2022, at 4:54 PM, Brian Burkhalter wrote: >> > > In our CI system `test/jdk/java/nio/file/WatchService/Move.java` appears to time out consistently, although I have not seen it happen on my local machine. > > ------------- > > PR: https://git.openjdk.org/jdk/pull/10140 For me. move subtree elapsed 1035 move file elapsed 2018 Elapsed times in millis -------------- next part -------------- An HTML attachment was scrubbed... URL: From tsteele at openjdk.org Thu Sep 8 13:53:34 2022 From: tsteele at openjdk.org (Tyler Steele) Date: Thu, 8 Sep 2022 13:53:34 GMT Subject: RFR: 8288627: [AIX] Implement WatchService using system library In-Reply-To: <3Vr3_Xpkj94Gq6-WqZPwGOWhRMApWqh0TMEFjGRavl4=.4344fb2d-7ca6-4021-9088-34e8ff376dfd@github.com> References: <3Vr3_Xpkj94Gq6-WqZPwGOWhRMApWqh0TMEFjGRavl4=.4344fb2d-7ca6-4021-9088-34e8ff376dfd@github.com> Message-ID: On Fri, 24 Jun 2022 15:17:47 GMT, Tyler Steele wrote: > This PR begins an effort to re-implement the WatchService API on AIX using 'AIX Event Infrastructure' (AHAFS). The initial motivation for doing so was to address errors found by some internal testing in the implementation based on PollingWatchService.java. > > I am submitting these changes before they are fully complete because (1) it represents a fairly large change that mostly works well, and could easily be improved upon in the future; (2) to get some early feedback on the changes so the final review process is quicker. My expected plan is to merge these changes after review, add any remaining test failures to a problem list, and return to them soon. > > ### Testing > I am waiting for confirmation that this re-implementation passes the internal tests we saw failing. Initial results look promising. In addition, I see the following failing tests from `test/jdk/java/nio/file/WatchService`: > > - Basic.java fails at testTwoWatchers. Having two WatchService instances registered with the same file, will require some additional work to be supported with AHAFS. This is currently a limitation of this implementation. > - DeleteInterference.java fails for the same reason as above. > - UpdateInterference.java fails for no good reason that I can deduce. Logging the test's progress shows that it doesn't seem to leave [the main loop](https://github.com/openjdk/jdk/blob/master/test/jdk/java/nio/file/WatchService/UpdateInterference.java#L97-L111), but that it makes is to within 1ms of doing so. > This pull request has been inactive for more than 4 weeks I still plan to return to this PR. ------------- PR: https://git.openjdk.org/jdk/pull/9281 From bpb at openjdk.org Thu Sep 8 14:53:40 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 8 Sep 2022 14:53:40 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: References: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> Message-ID: On Thu, 8 Sep 2022 07:21:07 GMT, Maxim Kartashev wrote: > Can you provide the `.jtr` file, please? I haven't seen the test hang, but by inserting different delays different systems can expose synchronization issues, of course. I can't give the whole .jtr file due to internal content, but here are some excerpts. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From brian.burkhalter at oracle.com Thu Sep 8 23:11:04 2022 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 8 Sep 2022 23:11:04 +0000 Subject: RFR: 8264744: (fs) Use file cloning in Linux version of Files::copy method [v13] In-Reply-To: References: Message-ID: On Sep 6, 2022, at 1:04 PM, Mike Hearn > wrote: Thank you! I'm the one who filed the original webbug report (some day I must figure out how to get direct access to the JDK bug tracker). This wasn't a theoretical request, it was motivated by a real product that my company has just launched called [Conveyor](https://conveyor.hydraulic.dev/). Conveyor is a sort of jpackage++ and is structured as a parallel build system in which intermediate steps are cached. The different tasks sometimes have to copy large files around (like JDKs, JARs etc) and using kernel level COW cloning will yield a large boost to performance on filesystems that support it. You?re welcome! We are pleased that you are happy with the change. If eventually you have any performance improvement to report, subjective or objective, please post it to nio-dev. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpb at openjdk.org Thu Sep 8 23:17:38 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 8 Sep 2022 23:17:38 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v2] In-Reply-To: References: Message-ID: <_hAyD7ck1ZCzQ6V10Czckg-tWcNLM_QYTA0VW_IpFcY=.043a53f6-921b-4555-b533-6f37f4f28db2@github.com> > Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8293122: Revert BsdFileStore; move volume capability check to BsdNativeDispatcher ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10188/files - new: https://git.openjdk.org/jdk/pull/10188/files/0084d916..a7917d22 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10188&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10188&range=00-01 Stats: 156 lines in 6 files changed: 52 ins; 99 del; 5 mod Patch: https://git.openjdk.org/jdk/pull/10188.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10188/head:pull/10188 PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Thu Sep 8 23:17:39 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 8 Sep 2022 23:17:39 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v2] In-Reply-To: References: Message-ID: On Wed, 7 Sep 2022 15:04:32 GMT, Alan Bateman wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8293122: Revert BsdFileStore; move volume capability check to BsdNativeDispatcher > > src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 101: > >> 99: BsdFileStore bfs = (BsdFileStore)provider().getFileStore(src); >> 100: if (!bfs.equals(provider().getFileStore(dst.getParent())) || >> 101: !bfs.supportsCloning()) > > Is 'dst' a relative path, in which case you can't rely on getParent. In any case, I don't think we should be checking the FileStore here as that means finding the mount entry. If we do clonefile then I think you'll have to rely on the EXDEV error. Fixed in a7917d2216f111d2aa2d3190c927d5897c826da1. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From alanb at openjdk.org Fri Sep 9 07:25:41 2022 From: alanb at openjdk.org (Alan Bateman) Date: Fri, 9 Sep 2022 07:25:41 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v2] In-Reply-To: References: Message-ID: On Wed, 7 Sep 2022 00:51:40 GMT, Brian Burkhalter wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8293122: Revert BsdFileStore; move volume capability check to BsdNativeDispatcher > > src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 110: > >> 108: switch (x.errno()) { >> 109: case ENOTSUP: // cloning not supported by filesystem >> 110: return IOStatus.UNSUPPORTED; > > It might be that `ENOTSUP` should instead result in `UNSUPPORTED_CASE` and that the static variable `cloneFileNotSupported` is unnecessary. Yes, ENOTSUP depends on the parameters to I think it maps to UNSUPPORTED_CASE. Given that it has to handle ENOTSUP then it may be that the check for the VOL_CAP_INT_CLONE feature isn't needed. You may have to compare to the performance of getattrlist to see if it helps when the file systems don't support clone. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From alanb at openjdk.org Fri Sep 9 07:29:43 2022 From: alanb at openjdk.org (Alan Bateman) Date: Fri, 9 Sep 2022 07:29:43 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v2] In-Reply-To: <_hAyD7ck1ZCzQ6V10Czckg-tWcNLM_QYTA0VW_IpFcY=.043a53f6-921b-4555-b533-6f37f4f28db2@github.com> References: <_hAyD7ck1ZCzQ6V10Czckg-tWcNLM_QYTA0VW_IpFcY=.043a53f6-921b-4555-b533-6f37f4f28db2@github.com> Message-ID: <-tC4t_MHkm1nBfDl3kmkqytRVYaHbg-SlhjactJdNpA=.e9285efe-8d05-4a55-850f-64cd5b23708f@github.com> On Thu, 8 Sep 2022 23:17:38 GMT, Brian Burkhalter wrote: >> Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293122: Revert BsdFileStore; move volume capability check to BsdNativeDispatcher src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 149: > 147: } catch (UnixException x) { > 148: if (flags.failIfUnableToCopyPosix) > 149: x.rethrowAsIOException(target); Can you compare this to the Unix implementation as I thought there was a best attempt to unlink when throwing an exception? ------------- PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Fri Sep 9 19:39:37 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 9 Sep 2022 19:39:37 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v3] In-Reply-To: References: Message-ID: > Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8293122: remove volatile cloneFileNotSupported; unlink target on failure; remove getattrlist ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10188/files - new: https://git.openjdk.org/jdk/pull/10188/files/a7917d22..f51cd186 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10188&range=02 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10188&range=01-02 Stats: 80 lines in 4 files changed: 6 ins; 69 del; 5 mod Patch: https://git.openjdk.org/jdk/pull/10188.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10188/head:pull/10188 PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Fri Sep 9 19:46:44 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 9 Sep 2022 19:46:44 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v3] In-Reply-To: References: Message-ID: On Fri, 9 Sep 2022 07:23:39 GMT, Alan Bateman wrote: >> src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 110: >> >>> 108: switch (x.errno()) { >>> 109: case ENOTSUP: // cloning not supported by filesystem >>> 110: return IOStatus.UNSUPPORTED; >> >> It might be that `ENOTSUP` should instead result in `UNSUPPORTED_CASE` and that the static variable `cloneFileNotSupported` is unnecessary. > > Yes, ENOTSUP depends on the parameters to I think it maps to UNSUPPORTED_CASE. Given that it has to handle ENOTSUP then it may be that the check for the VOL_CAP_INT_CLONE feature isn't needed. You may have to compare the performance of getattrlist to see if it helps when the file systems don't support clone. ENOTSUP is mapped to UNSUPPORTED_CASE in f51cd186fe624d03ce7c6a4f49140998e2d48322, which also removes getattrlist(). Benchmark testing in Java was inconclusive, but testing C code showed an improvement of only 1 microsecond for checking VOL_CAP_INT_CLONE on a volume which does not supporting cloning as opposed to just letting clonefile() set ENOTSUP. The extra code for getattrlist() does not seem to be worth it. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Fri Sep 9 19:46:44 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 9 Sep 2022 19:46:44 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v3] In-Reply-To: <6-x5WK0eFs1tAM4xeDxE-4FLhalshmUbZq7V70f6GKU=.6cb07e7d-6264-4086-b455-f9563dba0048@github.com> References: <6-x5WK0eFs1tAM4xeDxE-4FLhalshmUbZq7V70f6GKU=.6cb07e7d-6264-4086-b455-f9563dba0048@github.com> Message-ID: <9eaFvaz-Tv_aLBII_y-ipqeWTyMGvSFLB_I1BtYW8-Y=.63d499b9-a10a-480a-b429-38b6456266a2@github.com> On Wed, 7 Sep 2022 17:00:23 GMT, Brian Burkhalter wrote: >> src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 141: >> >>> 139: // cancellation is not possible, or attributes are not to be copied >>> 140: if (!cloneFileNotSupported && addressToPollForCancel == 0 && >>> 141: flags.copyPosixAttributes) { >> >> This will need some thought as this means it will only be used when Files.copy is invoked with the COPY_ATTRIBUTES option. > > `clonefile` appears to copy all attributes so if it is acceptable for that to occur if `COPY_ATTRIBUTES` is not set, then the `flags.copyPosixAttributes` check could be removed. My opinion is that the improved throughput outweighs any detriment from copying attributes when COPY_ATTRIBUTES is not specified and so the flags.copyPosixAttributes check should be removed. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Fri Sep 9 19:46:47 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 9 Sep 2022 19:46:47 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v2] In-Reply-To: <-tC4t_MHkm1nBfDl3kmkqytRVYaHbg-SlhjactJdNpA=.e9285efe-8d05-4a55-850f-64cd5b23708f@github.com> References: <_hAyD7ck1ZCzQ6V10Czckg-tWcNLM_QYTA0VW_IpFcY=.043a53f6-921b-4555-b533-6f37f4f28db2@github.com> <-tC4t_MHkm1nBfDl3kmkqytRVYaHbg-SlhjactJdNpA=.e9285efe-8d05-4a55-850f-64cd5b23708f@github.com> Message-ID: On Fri, 9 Sep 2022 07:25:57 GMT, Alan Bateman wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8293122: Revert BsdFileStore; move volume capability check to BsdNativeDispatcher > > src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 149: > >> 147: } catch (UnixException x) { >> 148: if (flags.failIfUnableToCopyPosix) >> 149: x.rethrowAsIOException(target); > > Can you compare this to the Unix implementation as I thought there was a best attempt to unlink when throwing an exception? You are correct: thanks for catching my gaffe. Fixed in f51cd186fe624d03ce7c6a4f49140998e2d48322. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From alanb at openjdk.org Sat Sep 10 14:12:44 2022 From: alanb at openjdk.org (Alan Bateman) Date: Sat, 10 Sep 2022 14:12:44 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v3] In-Reply-To: References: Message-ID: <_Jm67f21KxJ4zY1RCAeHZRrjnFol7dBsGtAOSfCbLRg=.943fd5bf-212d-4710-970a-169b8d7b3bbd@github.com> On Fri, 9 Sep 2022 19:39:37 GMT, Brian Burkhalter wrote: >> Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293122: remove volatile cloneFileNotSupported; unlink target on failure; remove getattrlist src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 100: > 98: try { > 99: unlink(dst); > 100: } catch (UnixException ignore) { } My previous comment about the missing unlink was a comment on BsdFileSystem.copyFile where it will leave dst in place when the chown fails. The unlink you've added to clone is harmless but the right place is copyFile so that it attempts to cleanup/unlink when an exception is thrown. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From alanb at openjdk.org Sun Sep 11 18:24:48 2022 From: alanb at openjdk.org (Alan Bateman) Date: Sun, 11 Sep 2022 18:24:48 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: References: Message-ID: On Fri, 2 Sep 2022 07:54:32 GMT, Maxim Kartashev wrote: > This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. > > Some things of note: > * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. > * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). > * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). > * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). > * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. > > This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. > > ### Testing > > * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). > * Also verified that new tests pass on Linux and Windows. > * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. I did a first pass over this. It's a good start and demonstrates that the file system events / FSEventStream* API can be used to implement WatchService. It's really unfortunate that it requires calling CFRunLoopRun to do the run loop as that means an upcall and JNI code that we would normally try to keep out of this area. One concern is that there is a lot of UnixPath (bytes) <-> String <-> CFString conversions going on and these will need time to work through (even with UTF-8). Also CFRunLoopThread will need to be an InnocuousThread to avoid inheriting inheritable thread locals or context class loader. One architectural issue is that the WatchService implementation should not be using Path.of or Files.* methods as that will cause a loop when interposing on the default provider. The provider implementation should only use use the Unix provider methods directly. The use of toRealPath().normalize() looks odd, I would expect the normalize to be a no-op here. This is an area that will require careful study as it's just not clear how the macOS work when the sym link is to another directory on another file system. Look at ptr_to_jlong and jlong_to_jptr to go between points and jlong and that should eliminate some of the casts. The code includes SUBTREE support. I had hope this wouldn't be included initially as it adds a bit more complexity and could be added later. Style-wise the code is very different to the existing code and I think we'll need to do a few cleanup passes (overly long lines, naming, formatting, overdue of final, ... all minor stuff and there is a lot to go through before these things). ------------- PR: https://git.openjdk.org/jdk/pull/10140 From mkartashev at openjdk.org Mon Sep 12 08:17:44 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Mon, 12 Sep 2022 08:17:44 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: References: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> Message-ID: <7VgVxVrHAjhC3ApCq0OJLcCowBEgkyETUtDs7B1-wy0=.ec645f0c-3486-4146-acb8-baa1684f7a24@github.com> On Thu, 8 Sep 2022 14:49:52 GMT, Brian Burkhalter wrote: >>> In our CI system `test/jdk/java/nio/file/WatchService/Move.java` appears to time out consistently, although I have not seen it happen on my local machine. >> >> Can you provide the `.jtr` file, please? I haven't seen the test hang, but by inserting different delays different systems can expose synchronization issues, of course. > >> Can you provide the `.jtr` file, please? I haven't seen the test hang, but by inserting different delays different systems can expose synchronization issues, of course. > > I can't give the whole .jtr file due to internal content, but here are some excerpts. @bplb Thanks for the log. It's really strange, though, that there is nothing from `System.out` because the test itself logs intensively even before any `WatchService` code is executed. Also, AFAIK timeouts trigger thread dump, which would be quite helpful. Are you not at liberty to share those as well? ------------- PR: https://git.openjdk.org/jdk/pull/10140 From mik3hall at gmail.com Mon Sep 12 11:24:23 2022 From: mik3hall at gmail.com (Michael Hall) Date: Mon, 12 Sep 2022 06:24:23 -0500 Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: <7VgVxVrHAjhC3ApCq0OJLcCowBEgkyETUtDs7B1-wy0=.ec645f0c-3486-4146-acb8-baa1684f7a24@github.com> References: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> <7VgVxVrHAjhC3ApCq0OJLcCowBEgkyETUtDs7B1-wy0=.ec645f0c-3486-4146-acb8-baa1684f7a24@github.com> Message-ID: <21F204C8-FF28-4E5D-AFBF-264C1074DF98@gmail.com> > On Sep 12, 2022, at 3:17 AM, Maxim Kartashev wrote: > > On Thu, 8 Sep 2022 14:49:52 GMT, Brian Burkhalter wrote: > >>>> In our CI system `test/jdk/java/nio/file/WatchService/Move.java` appears to time out consistently, although I have not seen it happen on my local machine. >>> >>> Can you provide the `.jtr` file, please? I haven't seen the test hang, but by inserting different delays different systems can expose synchronization issues, of course. >> >>> Can you provide the `.jtr` file, please? I haven't seen the test hang, but by inserting different delays different systems can expose synchronization issues, of course. >> >> I can't give the whole .jtr file due to internal content, but here are some excerpts. > > @bplb Thanks for the log. It's really strange, though, that there is nothing from `System.out` because the test itself logs intensively even before any `WatchService` code is executed. Also, AFAIK timeouts trigger thread dump, which would be quite helpful. Are you not at liberty to share those as well? > > ------------- > > PR: https://git.openjdk.org/jdk/pull/10140 Excuse my unofficial interest. But I had attempted a kqueue based OS/X watch service sometime back that failed, so it?s interesting to see one working. Mine would stop receiving kernel events running the LotsOfEvents test. It never got past that. Checking LotsOfEvents out of curiosity I notice it?s millis timing is? overflow elapsed 32200 queuing elapsed 24292 While again the Move test runs in like? move subtree elapsed 1048 move file elapsed 2018 So why would tests consistently time out in Move runs when LotsOfEvents takes longer for me? I am of course unfamiliar with the JDK testing but this seems odd. From bpb at openjdk.org Mon Sep 12 16:22:39 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Mon, 12 Sep 2022 16:22:39 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v4] In-Reply-To: References: Message-ID: > Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8293122: Move rollback unlink to copyFile method ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10188/files - new: https://git.openjdk.org/jdk/pull/10188/files/f51cd186..96db8562 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10188&range=03 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10188&range=02-03 Stats: 24 lines in 1 file changed: 9 ins; 5 del; 10 mod Patch: https://git.openjdk.org/jdk/pull/10188.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10188/head:pull/10188 PR: https://git.openjdk.org/jdk/pull/10188 From alanb at openjdk.org Mon Sep 12 16:46:40 2022 From: alanb at openjdk.org (Alan Bateman) Date: Mon, 12 Sep 2022 16:46:40 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v4] In-Reply-To: References: Message-ID: On Mon, 12 Sep 2022 16:22:39 GMT, Brian Burkhalter wrote: >> Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293122: Move rollback unlink to copyFile method Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.org/jdk/pull/10188 From mkartashev at openjdk.org Mon Sep 12 17:08:00 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Mon, 12 Sep 2022 17:08:00 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v2] In-Reply-To: References: Message-ID: > This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. > > Some things of note: > * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. > * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). > * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). > * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). > * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. > > This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. > > ### Testing > > * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). > * Also verified that new tests pass on Linux and Windows. > * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. Maxim Kartashev has updated the pull request incrementally with five additional commits since the last revision: - Removed support for recursive watch - Reformatted native code and removed jlong and pointer casts - Removed the dependency on provider(Path) - More robust way of obtaining run loop reference - CFRunLoopThread to use InnocuousThread ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10140/files - new: https://git.openjdk.org/jdk/pull/10140/files/e56e0bba..5b06c378 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=00-01 Stats: 407 lines in 3 files changed: 70 ins; 236 del; 101 mod Patch: https://git.openjdk.org/jdk/pull/10140.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10140/head:pull/10140 PR: https://git.openjdk.org/jdk/pull/10140 From mkartashev at openjdk.org Mon Sep 12 17:12:46 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Mon, 12 Sep 2022 17:12:46 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: References: Message-ID: On Sun, 11 Sep 2022 18:20:57 GMT, Alan Bateman wrote: >> This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. >> >> Some things of note: >> * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. >> * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). >> * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). >> * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). >> * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. >> >> This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. >> >> ### Testing >> >> * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). >> * Also verified that new tests pass on Linux and Windows. >> * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. > > I did a first pass over this. It's a good start and demonstrates that the file system events / FSEventStream* API can be used to implement WatchService. > > It's really unfortunate that it requires calling CFRunLoopRun to do the run loop as that means an upcall and JNI code that we would normally try to keep out of this area. One concern is that there is a lot of UnixPath (bytes) <-> String <-> CFString conversions going on and these will need time to work through (even with UTF-8). Also > CFRunLoopThread will need to be an InnocuousThread to avoid inheriting inheritable thread locals or context class loader. > > One architectural issue is that the WatchService implementation should not be using Path.of or Files.* methods as that will cause a loop when interposing on the default provider. The provider implementation should only use use the Unix provider methods directly. > > The use of toRealPath().normalize() looks odd, I would expect the normalize to be a no-op here. This is an area that will require careful study as it's just not clear how the macOS work when the sym link is to another directory on another file system. > > Look at ptr_to_jlong and jlong_to_jptr to go between points and jlong and that should eliminate some of the casts. > > The code includes SUBTREE support. I had hope this wouldn't be included initially as it adds a bit more complexity and could be added later. > > Style-wise the code is very different to the existing code and I think we'll need to do a few cleanup passes (overly long lines, naming, formatting, overdue of final, ... all minor stuff and there is a lot to go through before these things). @AlanBateman Thank you for taking time to review this, much appreciated. I made some changes based on your feedback; see inlined comments below. > It's really unfortunate that it requires calling CFRunLoopRun to do the run loop as that means an upcall and JNI code that we would normally try to keep out of this area. One concern is that there is a lot of UnixPath (bytes) <-> String <-> CFString conversions going on and these will need time to work through (even with UTF-8). Also CFRunLoopThread will need to be an InnocuousThread to avoid inheriting inheritable thread locals or context class loader. Done, `CFRunLoopThread` is now `InnocuousThread`. > One architectural issue is that the WatchService implementation should not be using Path.of or Files.* methods as that will cause a loop when interposing on the default provider. The provider implementation should only use use the Unix provider methods directly. Duly noted and, I think, corrected. > The use of toRealPath().normalize() looks odd, I would expect the normalize to be a no-op here. This is an area that will require careful study as it's just not clear how the macOS work when the sym link is to another directory on another file system. I may be misinterpreting you here, but to me it seems `toRealPath()` is actually necessary. Let's say the watch root is a directory named "../symlink" that points to "/Users/maxim/work/tests/dir-watcher/". The FSEvents API will report strings denoting absolute path names like "/Users/maxim/work/tests/dir-watcher/subdir" and in order to be able to remove the common prefix (watch root), one has to make that watch root into an unambiguous absolute path name. > Look at ptr_to_jlong and jlong_to_jptr to go between points and jlong and that should eliminate some of the casts. All such cases were eliminated. > The code includes SUBTREE support. I had hope this wouldn't be included initially as it adds a bit more complexity and could be added later. The recursive subdirectory watch support was removed (for now); let's focus on the initial support for FSEvents. > > Style-wise the code is very different to the existing code and I think we'll need to do a few cleanup passes (overly long lines, naming, formatting, overdue of final, ... all minor stuff and there is a lot to go through before these things). I also reformatted the native code and a bit of Java mostly to shorten the lines. The rest will have to be on case-by-case basis, I'm afraid. This time around I only ran the `test/jdk/java/nio/file/WatchService` tests on MacOs 10.15. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From mkartashev at openjdk.org Tue Sep 13 08:10:53 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Tue, 13 Sep 2022 08:10:53 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v3] In-Reply-To: References: Message-ID: > This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. > > Some things of note: > * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. > * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). > * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). > * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). > * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. > > This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. > > ### Testing > > * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). > * Also verified that new tests pass on Linux and Windows. > * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. Maxim Kartashev has updated the pull request incrementally with one additional commit since the last revision: Removed the change unnecessary without recursive watch ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10140/files - new: https://git.openjdk.org/jdk/pull/10140/files/5b06c378..94421827 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=02 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=01-02 Stats: 14 lines in 1 file changed: 0 ins; 13 del; 1 mod Patch: https://git.openjdk.org/jdk/pull/10140.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10140/head:pull/10140 PR: https://git.openjdk.org/jdk/pull/10140 From alanb at openjdk.org Tue Sep 13 11:09:44 2022 From: alanb at openjdk.org (Alan Bateman) Date: Tue, 13 Sep 2022 11:09:44 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v4] In-Reply-To: References: Message-ID: On Mon, 12 Sep 2022 16:22:39 GMT, Brian Burkhalter wrote: >> Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293122: Move rollback unlink to copyFile method src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 128: > 126: // Attempt to clone the source unless cancellation is not possible, > 127: // or attributes are not to be copied > 128: if (addressToPollForCancel == 0 && flags.copyPosixAttributes) { It wouldn't be wrong to the check for copyPosixAttributes here, in which case they would be copied by default (the spec allows for that). The only issue is that someone might become dependent on this and then be surprised when they code behaves differently when run on another platform. I'm curious if there is a way to test how clone behaves when it fails. What you have is okay although the unlink might only be needed when chown fails. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Tue Sep 13 16:21:40 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 13 Sep 2022 16:21:40 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v3] In-Reply-To: <_Jm67f21KxJ4zY1RCAeHZRrjnFol7dBsGtAOSfCbLRg=.943fd5bf-212d-4710-970a-169b8d7b3bbd@github.com> References: <_Jm67f21KxJ4zY1RCAeHZRrjnFol7dBsGtAOSfCbLRg=.943fd5bf-212d-4710-970a-169b8d7b3bbd@github.com> Message-ID: On Sat, 10 Sep 2022 14:09:21 GMT, Alan Bateman wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8293122: remove volatile cloneFileNotSupported; unlink target on failure; remove getattrlist > > src/java.base/macosx/classes/sun/nio/fs/BsdFileSystem.java line 100: > >> 98: try { >> 99: unlink(dst); >> 100: } catch (UnixException ignore) { } > > My previous comment about the missing unlink was a comment on BsdFileSystem.copyFile where it will leave dst in place when the chown fails. The unlink you've added to clone is harmless but the right place is copyFile so that it attempts to cleanup/unlink when an exception is thrown. Fixed in 96db8562fad4396500f8386c5f26c5fa98b7c373. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Tue Sep 13 16:27:56 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 13 Sep 2022 16:27:56 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v4] In-Reply-To: References: Message-ID: On Tue, 13 Sep 2022 11:05:56 GMT, Alan Bateman wrote: > The only issue is that someone might become dependent on this and then be surprised when they code behaves differently when run on another platform. This is a good reason to leave it as-is. > I'm curious if there is a way to test how clone behaves when it fails. What you have is okay although the unlink might only be needed when chown fails. All the described error numbers except EIO are things which one would expect the call to verify before actually starting to clone, so for those cases it would be unexpected to find that the destination exists after the failure. Provoking an EIO failure might be difficult. ------------- PR: https://git.openjdk.org/jdk/pull/10188 From brian.burkhalter at oracle.com Tue Sep 13 16:46:29 2022 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 13 Sep 2022 16:46:29 +0000 Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: <21F204C8-FF28-4E5D-AFBF-264C1074DF98@gmail.com> References: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> <7VgVxVrHAjhC3ApCq0OJLcCowBEgkyETUtDs7B1-wy0=.ec645f0c-3486-4146-acb8-baa1684f7a24@github.com> <21F204C8-FF28-4E5D-AFBF-264C1074DF98@gmail.com> Message-ID: <46842BA0-1DA7-48C2-88C9-C16F2A4A7AD9@oracle.com> On Sep 12, 2022, at 4:24 AM, Michael Hall > wrote: I had attempted a kqueue based OS/X watch service sometime back that failed, so it?s interesting to see one working. Concerning kqueue, please see my comment here: https://bugs.openjdk.org/browse/JDK-7133447?focusedCommentId=14522734&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14522734 Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpb at openjdk.org Tue Sep 13 16:46:51 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 13 Sep 2022 16:46:51 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: References: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> Message-ID: On Thu, 8 Sep 2022 14:49:52 GMT, Brian Burkhalter wrote: >>> In our CI system `test/jdk/java/nio/file/WatchService/Move.java` appears to time out consistently, although I have not seen it happen on my local machine. >> >> Can you provide the `.jtr` file, please? I haven't seen the test hang, but by inserting different delays different systems can expose synchronization issues, of course. > >> Can you provide the `.jtr` file, please? I haven't seen the test hang, but by inserting different delays different systems can expose synchronization issues, of course. > > I can't give the whole .jtr file due to internal content, but here are some excerpts. > @bplb Thanks for the log. It's really strange, though, that there is nothing from `System.out` because the test itself logs intensively even before any `WatchService` code is executed. Also, AFAIK timeouts trigger thread dump, which would be quite helpful. Are you not at liberty to share those as well? @mkartashev When a test times out, often not a lot of information is captured, including `System.out` and thread dumps. It can be frustrating to debug. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From bpb at openjdk.org Tue Sep 13 17:39:44 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 13 Sep 2022 17:39:44 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v5] In-Reply-To: References: Message-ID: > Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. Brian Burkhalter has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision: - Merge - 8293122: Move rollback unlink to copyFile method - 8293122: remove volatile cloneFileNotSupported; unlink target on failure; remove getattrlist - 8293122: Revert BsdFileStore; move volume capability check to BsdNativeDispatcher - 8293122: (fs) Use file cloning in macOS version of Files::copy method ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10188/files - new: https://git.openjdk.org/jdk/pull/10188/files/96db8562..afc9c675 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10188&range=04 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10188&range=03-04 Stats: 13721 lines in 304 files changed: 7974 ins; 4716 del; 1031 mod Patch: https://git.openjdk.org/jdk/pull/10188.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10188/head:pull/10188 PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Tue Sep 13 17:52:45 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 13 Sep 2022 17:52:45 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v3] In-Reply-To: References: Message-ID: On Tue, 13 Sep 2022 08:10:53 GMT, Maxim Kartashev wrote: >> This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. >> >> Some things of note: >> * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. >> * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). >> * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). >> * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). >> * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. >> >> This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. >> >> ### Testing >> >> * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). >> * Also verified that new tests pass on Linux and Windows. >> * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. > > Maxim Kartashev has updated the pull request incrementally with one additional commit since the last revision: > > Removed the change unnecessary without recursive watch I reran the `java/nio/file/WatchService` tests with the current code and there were no failures this time. I wonder whether this might be due to the change from `CFRunLoopThread` to `InnocuousThread`? I had some similar timeouts with the kqueue-based prototype I developed which I think were due to problems with thread cleanup. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From mik3hall at gmail.com Tue Sep 13 22:16:23 2022 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 13 Sep 2022 17:16:23 -0500 Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) In-Reply-To: <46842BA0-1DA7-48C2-88C9-C16F2A4A7AD9@oracle.com> References: <-HM3C22k7j9w8aIQ2DlFqmvIFvfIAStZvxrgEzzJGaA=.9f558890-86f5-426c-b341-6cb824a17368@github.com> <7VgVxVrHAjhC3ApCq0OJLcCowBEgkyETUtDs7B1-wy0=.ec645f0c-3486-4146-acb8-baa1684f7a24@github.com> <21F204C8-FF28-4E5D-AFBF-264C1074DF98@gmail.com> <46842BA0-1DA7-48C2-88C9-C16F2A4A7AD9@oracle.com> Message-ID: <8EA32A81-318F-417C-912F-3A1AF50907A5@gmail.com> > On Sep 13, 2022, at 11:46 AM, Brian Burkhalter wrote: > > >> On Sep 12, 2022, at 4:24 AM, Michael Hall > wrote: >> >> I had attempted a kqueue based OS/X watch service sometime back that failed, so it?s interesting to see one working. > > Concerning kqueue, please see my comment here: > > https://bugs.openjdk.org/browse/JDK-7133447?focusedCommentId=14522734&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14522734 > > Brian I suppose it could of been number of file descriptors. Do you remember specific problems with the LotsOfEvents test? I remember the java would block waiting for something to happen. The kqueue native thread would loop on it?s on thread waiting for events but the kernel would just stop sending them. It?s been a while since I looked at the code so I don?t remember if I had any cleanup that would release file descriptors. I probably should have. But I don?t remember on what conditions it ran if I did. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkartashev at openjdk.org Wed Sep 14 06:06:40 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Wed, 14 Sep 2022 06:06:40 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v3] In-Reply-To: References: Message-ID: On Tue, 13 Sep 2022 17:49:19 GMT, Brian Burkhalter wrote: > I reran the java/nio/file/WatchService tests with the current code and there were no failures this time. I wonder whether this might be due to the change from CFRunLoopThread to InnocuousThread? Thanks! Let's hope it stays this way. BTW, my initial version did have a thread cleanup problem that was caught by other tests. That was because I assumed the run loop would exit as soon as all input sources are pulled from it; that assumption proved to be false, so now there's `MacOSXWatchService.runLoopStop()`. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From alanb at openjdk.org Wed Sep 14 13:43:29 2022 From: alanb at openjdk.org (Alan Bateman) Date: Wed, 14 Sep 2022 13:43:29 GMT Subject: RFR: 8293122: (fs) Use file cloning in macOS version of Files::copy method [v5] In-Reply-To: References: Message-ID: On Tue, 13 Sep 2022 17:39:44 GMT, Brian Burkhalter wrote: >> Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. > > Brian Burkhalter has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision: > > - Merge > - 8293122: Move rollback unlink to copyFile method > - 8293122: remove volatile cloneFileNotSupported; unlink target on failure; remove getattrlist > - 8293122: Revert BsdFileStore; move volume capability check to BsdNativeDispatcher > - 8293122: (fs) Use file cloning in macOS version of Files::copy method Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.org/jdk/pull/10188 From bpb at openjdk.org Wed Sep 14 16:18:34 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 14 Sep 2022 16:18:34 GMT Subject: RFR: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux Message-ID: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Files in `/proc` advertise a size of zero for which direct and mapped transfers to not work. For this case, fall back to arbitrary channel transfer. ------------- Commit messages: - 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux Changes: https://git.openjdk.org/jdk/pull/10267/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10267&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8293502 Stats: 205 lines in 2 files changed: 191 ins; 0 del; 14 mod Patch: https://git.openjdk.org/jdk/pull/10267.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10267/head:pull/10267 PR: https://git.openjdk.org/jdk/pull/10267 From bpb at openjdk.org Wed Sep 14 20:15:48 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 14 Sep 2022 20:15:48 GMT Subject: Integrated: 8293122: (fs) Use file cloning in macOS version of Files::copy method In-Reply-To: References: Message-ID: On Wed, 7 Sep 2022 00:48:31 GMT, Brian Burkhalter wrote: > Use BSD system call `clonefile` if possible when using `Files::copy` on macOS. This pull request has now been integrated. Changeset: a75ddb83 Author: Brian Burkhalter URL: https://git.openjdk.org/jdk/commit/a75ddb836b2de0e75a65dbfa3b2f240db07a7d31 Stats: 124 lines in 5 files changed: 123 ins; 0 del; 1 mod 8293122: (fs) Use file cloning in macOS version of Files::copy method Reviewed-by: alanb ------------- PR: https://git.openjdk.org/jdk/pull/10188 From alanb at openjdk.org Thu Sep 15 11:09:48 2022 From: alanb at openjdk.org (Alan Bateman) Date: Thu, 15 Sep 2022 11:09:48 GMT Subject: RFR: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux In-Reply-To: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> References: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Message-ID: On Wed, 14 Sep 2022 16:00:17 GMT, Brian Burkhalter wrote: > Files in `/proc` advertise a size of zero for which direct and mapped transfers do not work. For this case, fall back to arbitrary channel transfer. src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 774: > 772: // Adjust count only if remaining > 0, i.e., > 773: // sz > position which means sz > 0 > 774: if (remaining < count && remaining > 0) It might be a bit more readable to swap this to test `remaining > 0 && remaining < count`. src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 949: > 947: // System calls supporting fast transfers might not work on files > 948: // which advertise zero size such as those in Linux /proc > 949: if (src instanceof FileChannelImpl fci && fci.size() > 0) { This means an extra fstat per transferFrom but maybe you've measured it with small files and it's okay? test/jdk/java/nio/file/Files/CopyProcFile.java line 52: > 50: */ > 51: public class CopyProcFile { > 52: static final String SOURCE = "/proc/mounts"; I wonder how stable this is. If the mount tables change while the test is running then I assume it will fail. test/jdk/java/nio/file/Files/CopyProcFile.java line 80: > 78: static long copy(String src, String dst) { > 79: try { > 80: return Files.size(Files.copy(Path.of(src), Path.of(dst))); I think it might be clearer if you split this into Path target = Files.copy(Path.of(src), Path.of(dst)); return Files.size(target); only because it might not be obvious that it's the target Path that is returned. test/jdk/java/nio/file/Files/CopyProcFile.java line 100: > 98: try (FileChannel fci = FileChannel.open(Path.of(src), READ); > 99: FileChannel fco = FileChannel.open(Path.of(dst), CREATE_NEW, > 100: WRITE);) { I don't know if this line break is intentional or not but it would be more readable if WRITE were moved to the previous line. Update: Same thing in transferFrom. ------------- PR: https://git.openjdk.org/jdk/pull/10267 From bpb at openjdk.org Thu Sep 15 15:43:50 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 15 Sep 2022 15:43:50 GMT Subject: RFR: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux In-Reply-To: References: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Message-ID: On Thu, 15 Sep 2022 11:05:42 GMT, Alan Bateman wrote: >> Files in `/proc` advertise a size of zero for which direct and mapped transfers do not work. For this case, fall back to arbitrary channel transfer. > > test/jdk/java/nio/file/Files/CopyProcFile.java line 52: > >> 50: */ >> 51: public class CopyProcFile { >> 52: static final String SOURCE = "/proc/mounts"; > > I wonder how stable this is. If the mount tables change while the test is running then I assume it will fail. Yes, I thought of that as well. Do you have any suggestion for an alternative? ------------- PR: https://git.openjdk.org/jdk/pull/10267 From bpb at openjdk.org Thu Sep 15 18:05:43 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 15 Sep 2022 18:05:43 GMT Subject: RFR: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux In-Reply-To: References: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Message-ID: On Thu, 15 Sep 2022 15:41:36 GMT, Brian Burkhalter wrote: >> test/jdk/java/nio/file/Files/CopyProcFile.java line 52: >> >>> 50: */ >>> 51: public class CopyProcFile { >>> 52: static final String SOURCE = "/proc/mounts"; >> >> I wonder how stable this is. If the mount tables change while the test is running then I assume it will fail. > > Yes, I thought of that as well. Do you have any suggestion for an alternative? Perhaps just using a small prefix of the mount tables would be more stable. ------------- PR: https://git.openjdk.org/jdk/pull/10267 From bpb at openjdk.org Thu Sep 15 19:19:41 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 15 Sep 2022 19:19:41 GMT Subject: RFR: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux In-Reply-To: References: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Message-ID: On Thu, 15 Sep 2022 18:01:52 GMT, Brian Burkhalter wrote: >> Yes, I thought of that as well. Do you have any suggestion for an alternative? > > Perhaps just using a small prefix of the mount tables would be more stable. Thanks. Yes, I doubt CPUs are swapped as often as mounts are changed. ------------- PR: https://git.openjdk.org/jdk/pull/10267 From bpb at openjdk.org Thu Sep 15 20:07:54 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 15 Sep 2022 20:07:54 GMT Subject: RFR: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux In-Reply-To: References: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Message-ID: On Thu, 15 Sep 2022 11:00:02 GMT, Alan Bateman wrote: >> Files in `/proc` advertise a size of zero for which direct and mapped transfers do not work. For this case, fall back to arbitrary channel transfer. > > src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 949: > >> 947: // System calls supporting fast transfers might not work on files >> 948: // which advertise zero size such as those in Linux /proc >> 949: if (src instanceof FileChannelImpl fci && fci.size() > 0) { > > This means an extra fstat per transferFrom but maybe you've measured it with small files and it's okay? It's about 20% slower for sizes 32 and 4096 (disk block size), and about 13% slower for size 32768. Instead making the direct and trusted methods fail independently does not help. ------------- PR: https://git.openjdk.org/jdk/pull/10267 From bpb at openjdk.org Thu Sep 15 23:07:47 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 15 Sep 2022 23:07:47 GMT Subject: RFR: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux [v2] In-Reply-To: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> References: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Message-ID: > Files in `/proc` advertise a size of zero for which direct and mapped transfers do not work. For this case, fall back to arbitrary channel transfer. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8293502: Use /proc/cpuinfo instead of /proc/mounts and other cleanup ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10267/files - new: https://git.openjdk.org/jdk/pull/10267/files/1b009421..95901194 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10267&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10267&range=00-01 Stats: 11 lines in 2 files changed: 3 ins; 2 del; 6 mod Patch: https://git.openjdk.org/jdk/pull/10267.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10267/head:pull/10267 PR: https://git.openjdk.org/jdk/pull/10267 From bpb at openjdk.org Thu Sep 15 23:07:50 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 15 Sep 2022 23:07:50 GMT Subject: RFR: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux [v2] In-Reply-To: References: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Message-ID: On Thu, 15 Sep 2022 10:59:20 GMT, Alan Bateman wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8293502: Use /proc/cpuinfo instead of /proc/mounts and other cleanup > > src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java line 774: > >> 772: // Adjust count only if remaining > 0, i.e., >> 773: // sz > position which means sz > 0 >> 774: if (remaining < count && remaining > 0) > > It might be a bit more readable to swap this to test `remaining > 0 && remaining < count`. Fixed in 959011949525bde73f36a702b59d1e27bec569c6. > test/jdk/java/nio/file/Files/CopyProcFile.java line 80: > >> 78: static long copy(String src, String dst) { >> 79: try { >> 80: return Files.size(Files.copy(Path.of(src), Path.of(dst))); > > I think it might be clearer if you split this into > > Path target = Files.copy(Path.of(src), Path.of(dst)); > return Files.size(target); > > only because it might not be obvious that it's the target Path that is returned. Fixed in 959011949525bde73f36a702b59d1e27bec569c6. > test/jdk/java/nio/file/Files/CopyProcFile.java line 100: > >> 98: try (FileChannel fci = FileChannel.open(Path.of(src), READ); >> 99: FileChannel fco = FileChannel.open(Path.of(dst), CREATE_NEW, >> 100: WRITE);) { > > I don't know if this line break is intentional or not but it would be more readable if WRITE were moved to the previous line. > > Update: Same thing in transferFrom. Fixed in 959011949525bde73f36a702b59d1e27bec569c6. ------------- PR: https://git.openjdk.org/jdk/pull/10267 From bpb at openjdk.org Thu Sep 15 23:07:51 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 15 Sep 2022 23:07:51 GMT Subject: RFR: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux [v2] In-Reply-To: References: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Message-ID: On Thu, 15 Sep 2022 19:15:41 GMT, Brian Burkhalter wrote: >> Perhaps just using a small prefix of the mount tables would be more stable. > > Thanks. Yes, I doubt CPUs are swapped as often as mounts are changed. Changed to `/proc/cpuinfo` in 959011949525bde73f36a702b59d1e27bec569c6. ------------- PR: https://git.openjdk.org/jdk/pull/10267 From alanb at openjdk.org Fri Sep 16 14:19:42 2022 From: alanb at openjdk.org (Alan Bateman) Date: Fri, 16 Sep 2022 14:19:42 GMT Subject: RFR: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux [v2] In-Reply-To: References: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Message-ID: On Thu, 15 Sep 2022 23:07:47 GMT, Brian Burkhalter wrote: >> Files in `/proc` advertise a size of zero for which direct and mapped transfers do not work. For this case, fall back to arbitrary channel transfer. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293502: Use /proc/cpuinfo instead of /proc/mounts and other cleanup Thanks for the updates, 95901194 looks good. ------------- Marked as reviewed by alanb (Reviewer). PR: https://git.openjdk.org/jdk/pull/10267 From bpb at openjdk.org Sat Sep 17 00:44:48 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Sat, 17 Sep 2022 00:44:48 GMT Subject: RFR: 8293331: Refactor FileChannelImpl into operating system-specific components [v3] In-Reply-To: References: Message-ID: > Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. Brian Burkhalter has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision: - 8293331: Split FileDispatcherImpl into operating system-specific components - Merge - 8293331: Move OS-specific code to FileDispatcher; remove OS-specific FileChannelImpl classes - 8293331: correct an error in the AIX FileChannelImpl.c - 8293331: Refactor FileChannelImpl into operating system-specific components ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10154/files - new: https://git.openjdk.org/jdk/pull/10154/files/4a3e5fc7..1af425b8 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=02 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=01-02 Stats: 60548 lines in 634 files changed: 31625 ins; 24471 del; 4452 mod Patch: https://git.openjdk.org/jdk/pull/10154.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10154/head:pull/10154 PR: https://git.openjdk.org/jdk/pull/10154 From alanb at openjdk.org Sun Sep 18 18:07:33 2022 From: alanb at openjdk.org (Alan Bateman) Date: Sun, 18 Sep 2022 18:07:33 GMT Subject: RFR: 8293331: Refactor FileChannelImpl into operating system-specific components [v3] In-Reply-To: References: Message-ID: On Sat, 17 Sep 2022 00:44:48 GMT, Brian Burkhalter wrote: >> Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. > > Brian Burkhalter has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision: > > - 8293331: Split FileDispatcherImpl into operating system-specific components > - Merge > - 8293331: Move OS-specific code to FileDispatcher; remove OS-specific FileChannelImpl classes > - 8293331: correct an error in the AIX FileChannelImpl.c > - 8293331: Refactor FileChannelImpl into operating system-specific components I took a first pass over the latest update and it looks quite good. I assume the issue and description should be changed now as this is about refactoring of FileDispatcherImpl rather than FileChannelImpl. Could FileChannelImpl.nd be a static field, there doesn't seen to be any reason to create one per FileChannel now. Also Unmapper shouldn't need its own now either. Can the static initialiser that calls IOUtil.load be removed from FileChannelImpl? The library loading is pushed down to FileDispatcherImpl now so I assume it is not needed. ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Mon Sep 19 16:00:54 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Mon, 19 Sep 2022 16:00:54 GMT Subject: Integrated: 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux In-Reply-To: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> References: <5TwJrI9447xc8lrWEdWYB7HgC5-GV922H4f-bYhs1Xw=.a2fba8b6-0414-4f7a-b988-6ca9d9b0d0c7@github.com> Message-ID: On Wed, 14 Sep 2022 16:00:17 GMT, Brian Burkhalter wrote: > Files in `/proc` advertise a size of zero for which direct and mapped transfers do not work. For this case, fall back to arbitrary channel transfer. This pull request has now been integrated. Changeset: 6e23b432 Author: Brian Burkhalter URL: https://git.openjdk.org/jdk/commit/6e23b43220a25532029a1d6b986a2249dd4e0e03 Stats: 206 lines in 2 files changed: 192 ins; 0 del; 14 mod 8293502: (fc) FileChannel::transfer methods fail to copy /proc files on Linux Reviewed-by: alanb ------------- PR: https://git.openjdk.org/jdk/pull/10267 From bpb at openjdk.org Mon Sep 19 18:54:44 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Mon, 19 Sep 2022 18:54:44 GMT Subject: RFR: 8293331: Refactor FileChannelImpl into operating system-specific components [v4] In-Reply-To: References: Message-ID: <1fZRIOg5zRrxY1YFLJXi_2p6YW3OnthRSBut7xp1JL8=.b4ee0bfd-67e3-4f36-9f61-5cb05950d493@github.com> > Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. Brian Burkhalter has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits: - Merge - 8293331: Clean up FileChannelImpl - 8293331: Split FileDispatcherImpl into operating system-specific components - Merge - 8293331: Move OS-specific code to FileDispatcher; remove OS-specific FileChannelImpl classes - 8293331: correct an error in the AIX FileChannelImpl.c - 8293331: Refactor FileChannelImpl into operating system-specific components ------------- Changes: https://git.openjdk.org/jdk/pull/10154/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=03 Stats: 2668 lines in 16 files changed: 1459 ins; 1188 del; 21 mod Patch: https://git.openjdk.org/jdk/pull/10154.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10154/head:pull/10154 PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Mon Sep 19 18:54:46 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Mon, 19 Sep 2022 18:54:46 GMT Subject: RFR: 8293331: Refactor FileChannelImpl into operating system-specific components [v3] In-Reply-To: References: Message-ID: On Sun, 18 Sep 2022 18:04:48 GMT, Alan Bateman wrote: > Could FileChannelImpl.nd be a static field, there doesn't seen to be any reason to create one per FileChannel now. Also Unmapper shouldn't need its own now either. > > Can the static initialiser that calls IOUtil.load be removed from FileChannelImpl? The library loading is pushed down to FileDispatcherImpl now so I assume it is not needed. Above all fixed in c6e271eb0d23f846d4c5ef55c3cb32ba2f7d896d. ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Tue Sep 20 01:51:37 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 20 Sep 2022 01:51:37 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v5] In-Reply-To: References: Message-ID: <0-4oZQSVom1lvqZ-9H9Z67Tiqfikd6Y2DNFiudNjFzw=.8d3484c9-d455-4c08-a749-6011863d8435@github.com> > Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8293331: Fix static initialization order on Windows ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10154/files - new: https://git.openjdk.org/jdk/pull/10154/files/3190a8b2..ff80c1f5 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=04 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=03-04 Stats: 15 lines in 1 file changed: 3 ins; 6 del; 6 mod Patch: https://git.openjdk.org/jdk/pull/10154.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10154/head:pull/10154 PR: https://git.openjdk.org/jdk/pull/10154 From alanb at openjdk.org Tue Sep 20 08:40:06 2022 From: alanb at openjdk.org (Alan Bateman) Date: Tue, 20 Sep 2022 08:40:06 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components In-Reply-To: References: Message-ID: On Sat, 3 Sep 2022 00:59:42 GMT, Brian Burkhalter wrote: > Although an AIX implementation is provided, it will have to be verified at a later date by someone with access to the requisite platform. @backwaterred Are you able to test the AIX changes? ------------- PR: https://git.openjdk.org/jdk/pull/10154 From alanb at openjdk.org Tue Sep 20 08:40:06 2022 From: alanb at openjdk.org (Alan Bateman) Date: Tue, 20 Sep 2022 08:40:06 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v5] In-Reply-To: <0-4oZQSVom1lvqZ-9H9Z67Tiqfikd6Y2DNFiudNjFzw=.8d3484c9-d455-4c08-a749-6011863d8435@github.com> References: <0-4oZQSVom1lvqZ-9H9Z67Tiqfikd6Y2DNFiudNjFzw=.8d3484c9-d455-4c08-a749-6011863d8435@github.com> Message-ID: <1T_UqkSKxQE4CKZxBl0AFw_24a8dUZ8wuiyWV6vYqrM=.b5811d00-db4e-4424-ad9a-7ebdcb7965a7@github.com> On Tue, 20 Sep 2022 01:51:37 GMT, Brian Burkhalter wrote: >> Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293331: Fix static initialization order on Windows The overall refactoring looks okay. Most of the change is code moving to another file so it's hard to see if there are any changes and it's hard to see if the list of include files is accurate. Everything I checked seemed okay and unchanged. ------------- Marked as reviewed by alanb (Reviewer). PR: https://git.openjdk.org/jdk/pull/10154 From tsteele at openjdk.org Tue Sep 20 16:22:42 2022 From: tsteele at openjdk.org (Tyler Steele) Date: Tue, 20 Sep 2022 16:22:42 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v3] In-Reply-To: References: Message-ID: On Mon, 19 Sep 2022 18:51:59 GMT, Brian Burkhalter wrote: >> I took a first pass over the latest update and it looks quite good. I assume the issue and description should be changed now as this is about refactoring of FileDispatcherImpl rather than FileChannelImpl. >> >> Could FileChannelImpl.nd be a static field, there doesn't seen to be any reason to create one per FileChannel now. Also Unmapper shouldn't need its own now either. >> >> Can the static initialiser that calls IOUtil.load be removed from FileChannelImpl? The library loading is pushed down to FileDispatcherImpl now so I assume it is not needed. > >> Could FileChannelImpl.nd be a static field, there doesn't seen to be any reason to create one per FileChannel now. Also Unmapper shouldn't need its own now either. >> >> Can the static initialiser that calls IOUtil.load be removed from FileChannelImpl? The library loading is pushed down to FileDispatcherImpl now so I assume it is not needed. > > Above all fixed in c6e271eb0d23f846d4c5ef55c3cb32ba2f7d896d. I am happy to test this on AIX @bplb & @AlanBateman. I will update this message when it completes. --- Testing in progress ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Tue Sep 20 16:47:46 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 20 Sep 2022 16:47:46 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v3] In-Reply-To: References: Message-ID: On Mon, 19 Sep 2022 18:51:59 GMT, Brian Burkhalter wrote: >> I took a first pass over the latest update and it looks quite good. I assume the issue and description should be changed now as this is about refactoring of FileDispatcherImpl rather than FileChannelImpl. >> >> Could FileChannelImpl.nd be a static field, there doesn't seen to be any reason to create one per FileChannel now. Also Unmapper shouldn't need its own now either. >> >> Can the static initialiser that calls IOUtil.load be removed from FileChannelImpl? The library loading is pushed down to FileDispatcherImpl now so I assume it is not needed. > >> Could FileChannelImpl.nd be a static field, there doesn't seen to be any reason to create one per FileChannel now. Also Unmapper shouldn't need its own now either. >> >> Can the static initialiser that calls IOUtil.load be removed from FileChannelImpl? The library loading is pushed down to FileDispatcherImpl now so I assume it is not needed. > > Above all fixed in c6e271eb0d23f846d4c5ef55c3cb32ba2f7d896d. > I am happy to test this on AIX @bplb & @AlanBateman. I will update this message when it completes. > > Testing in progress Thank you! ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Tue Sep 20 16:56:16 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 20 Sep 2022 16:56:16 GMT Subject: RFR: 8294068: Unconditional and eager load of nio library since JDK-8264744 Message-ID: Move `LinuxFileSystem` native code to `LinuxNativeDispatcher`. ------------- Commit messages: - 8294068: Unconditional and eager load of nio library since JDK-8264744 Changes: https://git.openjdk.org/jdk/pull/10362/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10362&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8294068 Stats: 263 lines in 4 files changed: 102 ins; 161 del; 0 mod Patch: https://git.openjdk.org/jdk/pull/10362.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10362/head:pull/10362 PR: https://git.openjdk.org/jdk/pull/10362 From alanb at openjdk.org Tue Sep 20 17:43:48 2022 From: alanb at openjdk.org (Alan Bateman) Date: Tue, 20 Sep 2022 17:43:48 GMT Subject: RFR: 8294068: Unconditional and eager load of nio library since JDK-8264744 In-Reply-To: References: Message-ID: On Tue, 20 Sep 2022 16:47:09 GMT, Brian Burkhalter wrote: > Move `LinuxFileSystem` native code to `LinuxNativeDispatcher`. Moving it to LinuxNativeDispatcher is better so I think the change is okay. That said, libnio will likely be loaded early in any significant program. ------------- Marked as reviewed by alanb (Reviewer). PR: https://git.openjdk.org/jdk/pull/10362 From redestad at openjdk.org Tue Sep 20 19:26:44 2022 From: redestad at openjdk.org (Claes Redestad) Date: Tue, 20 Sep 2022 19:26:44 GMT Subject: RFR: 8294068: Unconditional and eager load of nio library since JDK-8264744 In-Reply-To: References: Message-ID: On Tue, 20 Sep 2022 16:47:09 GMT, Brian Burkhalter wrote: > Move `LinuxFileSystem` native code to `LinuxNativeDispatcher`. LGTM ------------- Marked as reviewed by redestad (Reviewer). PR: https://git.openjdk.org/jdk/pull/10362 From bpb at openjdk.org Tue Sep 20 21:09:52 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 20 Sep 2022 21:09:52 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v3] In-Reply-To: References: Message-ID: On Tue, 13 Sep 2022 08:10:53 GMT, Maxim Kartashev wrote: >> This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. >> >> Some things of note: >> * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. >> * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). >> * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). >> * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). >> * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. >> >> This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. >> >> ### Testing >> >> * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). >> * Also verified that new tests pass on Linux and Windows. >> * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. > > Maxim Kartashev has updated the pull request incrementally with one additional commit since the last revision: > > Removed the change unnecessary without recursive watch src/java.base/macosx/classes/sun/nio/fs/MacOSXWatchService.java line 129: > 127: * that need to be re-scanned. > 128: */ > 129: private void callback(final long eventStreamRef, Could this be named something more descriptive than `callback`, perhaps even `handleEvents` even though there is a method of the same name defined in `MacOSXWatchKey`? src/java.base/macosx/classes/sun/nio/fs/MacOSXWatchService.java line 396: > 394: if (!relativeRootPath.equals(path)) { > 395: // Ignore events from subdirectories for now. > 396: continue; Should a line eventFlagsPtr += SIZEOF_FS_EVENT_STREAM_EVENT_FLAGS; be added here? (The definition of `SIZEOF_FS_EVENT_STREAM_EVENT_FLAGS` would have to be moved up.) src/java.base/macosx/classes/sun/nio/fs/MacOSXWatchService.java line 399: > 397: } > 398: > 399: final int flags = unsafe.getInt(eventFlagsPtr); This looks like the only use of `Unsafe`. Could the objective be accomplished without using it, e.g., changing `long eventFlagsPtr` to a `long[]` or, maybe even simpler, a `boolean[]`? src/java.base/macosx/native/libnio/fs/MacOSXWatchService.c line 73: > 71: (*env)->CallVoidMethod(env, watchService, callbackMID, > 72: streamRef, javaEventPathsArray, eventFlags); > 73: } Is there a way to notify of events without an upcall? Maybe add the necessary values to a queue of some sort observed by the Java layer? src/java.base/macosx/native/libnio/fs/MacOSXWatchService.c line 89: > 87: { > 88: JNIEnv *env = (JNIEnv *) JNU_GetEnv(jvm, JNI_VERSION_1_2); > 89: if (!env) { // Shouldn't happen as run loop starts from Java code Is this check, hence the `jvm` extern, really necessary? src/java.base/macosx/native/libnio/fs/MacOSXWatchService.c line 199: > 197: > 198: FSEventStreamStop(streamRef); // Unregister with the FS Events service. > 199: // No more callbacks from this stream. Does `FSEventStreamUnscheduleFromRunLoop()` need to be called here? ------------- PR: https://git.openjdk.org/jdk/pull/10140 From bpb at openjdk.org Tue Sep 20 22:06:39 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Tue, 20 Sep 2022 22:06:39 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v3] In-Reply-To: References: Message-ID: On Tue, 20 Sep 2022 16:18:51 GMT, Tyler Steele wrote: > test/jdk/jdk/nio... 27/27 passing @backwaterred If tests passed then there was no build issue so I will integrate this tomorrow. Thanks for testing! ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Wed Sep 21 00:49:50 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 21 Sep 2022 00:49:50 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v6] In-Reply-To: References: Message-ID: > Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8293331: Remove unneeded C header files ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10154/files - new: https://git.openjdk.org/jdk/pull/10154/files/ff80c1f5..a2da4deb Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=05 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=04-05 Stats: 19 lines in 3 files changed: 1 ins; 18 del; 0 mod Patch: https://git.openjdk.org/jdk/pull/10154.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10154/head:pull/10154 PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Wed Sep 21 00:49:51 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 21 Sep 2022 00:49:51 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v5] In-Reply-To: <1T_UqkSKxQE4CKZxBl0AFw_24a8dUZ8wuiyWV6vYqrM=.b5811d00-db4e-4424-ad9a-7ebdcb7965a7@github.com> References: <0-4oZQSVom1lvqZ-9H9Z67Tiqfikd6Y2DNFiudNjFzw=.8d3484c9-d455-4c08-a749-6011863d8435@github.com> <1T_UqkSKxQE4CKZxBl0AFw_24a8dUZ8wuiyWV6vYqrM=.b5811d00-db4e-4424-ad9a-7ebdcb7965a7@github.com> Message-ID: On Tue, 20 Sep 2022 08:37:18 GMT, Alan Bateman wrote: > [...] it's hard to see if the list of include files is accurate. Unneeded include files are removed in a2da4deba931fdb67e5bd8b5c34bb562490ace4e. I don't know whether the absence of those removed in `UnixFileDispactherImpl` will cause problems on AIX (@backwaterred please test the build again, thanks). ------------- PR: https://git.openjdk.org/jdk/pull/10154 From alanb at openjdk.org Wed Sep 21 06:18:46 2022 From: alanb at openjdk.org (Alan Bateman) Date: Wed, 21 Sep 2022 06:18:46 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v6] In-Reply-To: References: Message-ID: On Wed, 21 Sep 2022 00:49:50 GMT, Brian Burkhalter wrote: >> Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293331: Remove unneeded C header files The update removed the include of jni.h but JNI types are used. They get included by the javac generated header files but might be clearer to not remove it. ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Wed Sep 21 14:55:24 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 21 Sep 2022 14:55:24 GMT Subject: Integrated: 8294068: Unconditional and eager load of nio library since JDK-8264744 In-Reply-To: References: Message-ID: On Tue, 20 Sep 2022 16:47:09 GMT, Brian Burkhalter wrote: > Move `LinuxFileSystem` native code to `LinuxNativeDispatcher`. This pull request has now been integrated. Changeset: e1958971 Author: Brian Burkhalter URL: https://git.openjdk.org/jdk/commit/e1958971c95a651da9e7fca7a98828f7fd0391bb Stats: 263 lines in 4 files changed: 102 ins; 161 del; 0 mod 8294068: Unconditional and eager load of nio library since JDK-8264744 Reviewed-by: alanb, redestad ------------- PR: https://git.openjdk.org/jdk/pull/10362 From bpb at openjdk.org Wed Sep 21 15:00:44 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 21 Sep 2022 15:00:44 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v7] In-Reply-To: References: Message-ID: > Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8293331: Reinstate #include "jni.h" ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10154/files - new: https://git.openjdk.org/jdk/pull/10154/files/a2da4deb..32d042d4 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=06 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=05-06 Stats: 3 lines in 3 files changed: 3 ins; 0 del; 0 mod Patch: https://git.openjdk.org/jdk/pull/10154.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10154/head:pull/10154 PR: https://git.openjdk.org/jdk/pull/10154 From tsteele at openjdk.org Wed Sep 21 15:15:22 2022 From: tsteele at openjdk.org (Tyler Steele) Date: Wed, 21 Sep 2022 15:15:22 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v7] In-Reply-To: References: Message-ID: <_2Zp6sDzHWds7TCsNiCciuSg-1LBQTpFEH0Up8oofEc=.b00650ac-0177-4359-a7e3-e37acd02eb64@github.com> On Wed, 21 Sep 2022 15:00:44 GMT, Brian Burkhalter wrote: >> Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293331: Reinstate #include "jni.h" The tests are in progress :-) ------------- PR: https://git.openjdk.org/jdk/pull/10154 From alanb at openjdk.org Wed Sep 21 15:20:27 2022 From: alanb at openjdk.org (Alan Bateman) Date: Wed, 21 Sep 2022 15:20:27 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v7] In-Reply-To: References: Message-ID: On Wed, 21 Sep 2022 15:00:44 GMT, Brian Burkhalter wrote: >> Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293331: Reinstate #include "jni.h" Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.org/jdk/pull/10154 From tsteele at openjdk.org Wed Sep 21 16:00:17 2022 From: tsteele at openjdk.org (Tyler Steele) Date: Wed, 21 Sep 2022 16:00:17 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v5] In-Reply-To: References: <0-4oZQSVom1lvqZ-9H9Z67Tiqfikd6Y2DNFiudNjFzw=.8d3484c9-d455-4c08-a749-6011863d8435@github.com> <1T_UqkSKxQE4CKZxBl0AFw_24a8dUZ8wuiyWV6vYqrM=.b5811d00-db4e-4424-ad9a-7ebdcb7965a7@github.com> Message-ID: On Wed, 21 Sep 2022 00:45:52 GMT, Brian Burkhalter wrote: >> The overall refactoring looks okay. Most of the change is code moving to another file so it's hard to see if there are any changes and it's hard to see if the list of include files is accurate. Everything I checked seemed okay and unchanged. > >> [...] it's hard to see if the list of include files is accurate. > > Unneeded include files are removed in a2da4deba931fdb67e5bd8b5c34bb562490ace4e. I don't know whether the absence of those removed in `UnixFileDispactherImpl` will cause problems on AIX (@backwaterred please test the build again, thanks). Looking good @bplb. The jdk/nio tests are all passing. ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Wed Sep 21 16:00:18 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 21 Sep 2022 16:00:18 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v5] In-Reply-To: References: <0-4oZQSVom1lvqZ-9H9Z67Tiqfikd6Y2DNFiudNjFzw=.8d3484c9-d455-4c08-a749-6011863d8435@github.com> <1T_UqkSKxQE4CKZxBl0AFw_24a8dUZ8wuiyWV6vYqrM=.b5811d00-db4e-4424-ad9a-7ebdcb7965a7@github.com> Message-ID: <4Jl-hyO8B6bN6vXo-qoZ4BIm89hbEB3MIMJEet9m0Pw=.8fefe56f-2e0b-408c-b13d-63dde022b6a9@github.com> On Wed, 21 Sep 2022 00:45:52 GMT, Brian Burkhalter wrote: >> The overall refactoring looks okay. Most of the change is code moving to another file so it's hard to see if there are any changes and it's hard to see if the list of include files is accurate. Everything I checked seemed okay and unchanged. > >> [...] it's hard to see if the list of include files is accurate. > > Unneeded include files are removed in a2da4deba931fdb67e5bd8b5c34bb562490ace4e. I don't know whether the absence of those removed in `UnixFileDispactherImpl` will cause problems on AIX (@backwaterred please test the build again, thanks). > Looking good @bplb. The jdk/nio tests are all passing. Thank you, @backwaterred, for the assist. ------------- PR: https://git.openjdk.org/jdk/pull/10154 From tsteele at openjdk.org Wed Sep 21 16:03:17 2022 From: tsteele at openjdk.org (Tyler Steele) Date: Wed, 21 Sep 2022 16:03:17 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v5] In-Reply-To: References: <0-4oZQSVom1lvqZ-9H9Z67Tiqfikd6Y2DNFiudNjFzw=.8d3484c9-d455-4c08-a749-6011863d8435@github.com> <1T_UqkSKxQE4CKZxBl0AFw_24a8dUZ8wuiyWV6vYqrM=.b5811d00-db4e-4424-ad9a-7ebdcb7965a7@github.com> Message-ID: On Wed, 21 Sep 2022 15:56:52 GMT, Tyler Steele wrote: >>> [...] it's hard to see if the list of include files is accurate. >> >> Unneeded include files are removed in a2da4deba931fdb67e5bd8b5c34bb562490ace4e. I don't know whether the absence of those removed in `UnixFileDispactherImpl` will cause problems on AIX (@backwaterred please test the build again, thanks). > > Looking good @bplb. The jdk/nio tests are all passing. > Thank you, @backwaterred, for the assist. You're welcome ? ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Thu Sep 22 00:32:15 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 22 Sep 2022 00:32:15 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v8] In-Reply-To: References: Message-ID: > Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8293331: Add missing init() call in UnixFileDispatcherImpl static initializer ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10154/files - new: https://git.openjdk.org/jdk/pull/10154/files/32d042d4..b73c99d2 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=07 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10154&range=06-07 Stats: 1 line in 1 file changed: 1 ins; 0 del; 0 mod Patch: https://git.openjdk.org/jdk/pull/10154.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10154/head:pull/10154 PR: https://git.openjdk.org/jdk/pull/10154 From vtewari at openjdk.org Thu Sep 22 06:04:17 2022 From: vtewari at openjdk.org (Vyom Tewari) Date: Thu, 22 Sep 2022 06:04:17 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v8] In-Reply-To: References: Message-ID: <1n-gpAD-GQPUYUIFI7IPoLPo1jQQPk6orRVaxbVV88Q=.6123e39b-1051-44cb-a5af-edd38ff0ecdc@github.com> On Thu, 22 Sep 2022 00:32:15 GMT, Brian Burkhalter wrote: >> Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293331: Add missing init() call in UnixFileDispatcherImpl static initializer src/java.base/linux/classes/sun/nio/ch/FileDispatcherImpl.java line 51: > 49: // -- Native methods -- > 50: > 51: static native long transferTo0(FileDescriptor src, long position, can we make these native methods(transferTo0, transferFrom0 etc) to private access specifier, these methods are not called from outside ?. ------------- PR: https://git.openjdk.org/jdk/pull/10154 From jpai at openjdk.org Thu Sep 22 06:50:06 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Thu, 22 Sep 2022 06:50:06 GMT Subject: RFR: 8293792: runtime/Dictionary/ProtectionDomainCacheTest.java fails with FileAlreadyExistsException: /tmp Message-ID: Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8293792? As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. tier1, tier2 and tier3 tests passed with this change. ------------- Commit messages: - newline at end of test file - fix whitespace issue in test - 8293792: Follow symlinks in Files.createDirectories for checking file existence Changes: https://git.openjdk.org/jdk/pull/10383/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10383&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8293792 Stats: 79 lines in 2 files changed: 78 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jdk/pull/10383.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10383/head:pull/10383 PR: https://git.openjdk.org/jdk/pull/10383 From alanb at openjdk.org Thu Sep 22 06:52:29 2022 From: alanb at openjdk.org (Alan Bateman) Date: Thu, 22 Sep 2022 06:52:29 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v8] In-Reply-To: References: Message-ID: <-06a1TOfh5dei91ifDnplvjae7zMvwYIVQak-A1Qmtc=.e793086c-1c3a-4742-9d91-b7b6d0f0f8fc@github.com> On Thu, 22 Sep 2022 00:32:15 GMT, Brian Burkhalter wrote: >> Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8293331: Add missing init() call in UnixFileDispatcherImpl static initializer Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.org/jdk/pull/10154 From mkartashev at openjdk.org Thu Sep 22 07:10:25 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Thu, 22 Sep 2022 07:10:25 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v3] In-Reply-To: References: Message-ID: <4tbbedLVW5VADXjUbXB7CssAy6d4muCAaIPxIosfF9k=.5c003921-08d3-424a-a964-95522f415472@github.com> On Tue, 20 Sep 2022 20:59:50 GMT, Brian Burkhalter wrote: >> Maxim Kartashev has updated the pull request incrementally with one additional commit since the last revision: >> >> Removed the change unnecessary without recursive watch > > src/java.base/macosx/classes/sun/nio/fs/MacOSXWatchService.java line 399: > >> 397: } >> 398: >> 399: final int flags = unsafe.getInt(eventFlagsPtr); > > This looks like the only use of `Unsafe`. Could the objective be accomplished without using it, e.g., changing `long eventFlagsPtr` to a `long[]` or, maybe even simpler, a `boolean[]`? Earlier on the mailing list @AlanBateman wrote "It would help the review if the native code is just simple wrappers about the FSEvents as we'll likely replace a lot of the JNI code in this area in time." I understood that as a suggestion to keep the native code as simple as possible and, conversely, do as much as possible on the Java side. Reading native memory here was one of the decisions that were based on that understanding. I may have, of course, read too much into it, so I'm not opposed to implementing your suggestion. I would like, however, to have a confirmation that this direction of the changes is OK with @AlanBateman also. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From jiefu at openjdk.org Thu Sep 22 07:14:19 2022 From: jiefu at openjdk.org (Jie Fu) Date: Thu, 22 Sep 2022 07:14:19 GMT Subject: RFR: 8293792: runtime/Dictionary/ProtectionDomainCacheTest.java fails with FileAlreadyExistsException: /tmp In-Reply-To: References: Message-ID: <305O1zS9M5oVl8d1ZFELkya8FosvdyZj0vIFoY2oIhY=.53b2b3af-0290-47aa-9ae2-ed9b02f5ac23@github.com> On Thu, 22 Sep 2022 06:34:53 GMT, Jaikiran Pai wrote: > Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8293792? > > As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. > tier1, tier2 and tier3 tests passed with this change. It passed on our machine with this patch. Thanks for fixing it. ------------- PR: https://git.openjdk.org/jdk/pull/10383 From alanb at openjdk.org Thu Sep 22 07:45:28 2022 From: alanb at openjdk.org (Alan Bateman) Date: Thu, 22 Sep 2022 07:45:28 GMT Subject: RFR: 8293792: runtime/Dictionary/ProtectionDomainCacheTest.java fails with FileAlreadyExistsException: /tmp In-Reply-To: References: Message-ID: On Thu, 22 Sep 2022 06:34:53 GMT, Jaikiran Pai wrote: > Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8293792? > > As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. > tier1, tier2 and tier3 tests passed with this change. I think create a new issue for Files.createDirectories issue or else move the existing issue from hotspot/runtime and change the description. I plan to review this change but not today because it require digging into some security issues. For the test then I think it will mean moving the existing tests from Misc.java to the new source file. That will fit with the description that you propose for the new test. ------------- PR: https://git.openjdk.org/jdk/pull/10383 From mkartashev at openjdk.org Thu Sep 22 07:49:23 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Thu, 22 Sep 2022 07:49:23 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v3] In-Reply-To: References: Message-ID: On Tue, 20 Sep 2022 20:55:48 GMT, Brian Burkhalter wrote: >> Maxim Kartashev has updated the pull request incrementally with one additional commit since the last revision: >> >> Removed the change unnecessary without recursive watch > > src/java.base/macosx/classes/sun/nio/fs/MacOSXWatchService.java line 129: > >> 127: * that need to be re-scanned. >> 128: */ >> 129: private void callback(final long eventStreamRef, > > Could this be named something more descriptive than `callback`, perhaps even `handleEvents` even though there is a method of the same name defined in `MacOSXWatchKey`? Sure. > src/java.base/macosx/classes/sun/nio/fs/MacOSXWatchService.java line 396: > >> 394: if (!relativeRootPath.equals(path)) { >> 395: // Ignore events from subdirectories for now. >> 396: continue; > > Should a line > > eventFlagsPtr += SIZEOF_FS_EVENT_STREAM_EVENT_FLAGS; > > be added here? (The definition of `SIZEOF_FS_EVENT_STREAM_EVENT_FLAGS` would have to be moved up.) Yes, most certainly. Thanks for catching this! > src/java.base/macosx/native/libnio/fs/MacOSXWatchService.c line 89: > >> 87: { >> 88: JNIEnv *env = (JNIEnv *) JNU_GetEnv(jvm, JNI_VERSION_1_2); >> 89: if (!env) { // Shouldn't happen as run loop starts from Java code > > Is this check, hence the `jvm` extern, really necessary? The check isn't strictly necessary, so I'm goign to drop it (there's a JNI version check in `nio_util.c`), but the `jvm` extern seems necessary because that's how `JNIEnv` is obtained and the latter is necessary to communicate the events back to the Java side. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From mkartashev at openjdk.org Thu Sep 22 07:52:12 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Thu, 22 Sep 2022 07:52:12 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v3] In-Reply-To: References: Message-ID: On Tue, 20 Sep 2022 21:03:31 GMT, Brian Burkhalter wrote: >> Maxim Kartashev has updated the pull request incrementally with one additional commit since the last revision: >> >> Removed the change unnecessary without recursive watch > > src/java.base/macosx/native/libnio/fs/MacOSXWatchService.c line 73: > >> 71: (*env)->CallVoidMethod(env, watchService, callbackMID, >> 72: streamRef, javaEventPathsArray, eventFlags); >> 73: } > > Is there a way to notify of events without an upcall? Maybe add the necessary values to a queue of some sort observed by the Java layer? I'm not sure I know how to synchronize native code with Java without any upcall to the Java side; but maybe you are suggesting something as simple as a `notify()` call in place of the current callback with many arguments?. And also this goes against my understanding that the native part has to be kept as simple and as straightforward as possible. Having it maintain a queue that has to be synchronized with Java will certainly complicate the native part. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From mkartashev at openjdk.org Thu Sep 22 08:01:23 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Thu, 22 Sep 2022 08:01:23 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v3] In-Reply-To: References: Message-ID: <8t661Amwe6szfIXqD26DDR8HTdY3OJI5ibdH-o7a_BE=.dc2f788d-31d3-428f-ae56-b6e38f9bb3ee@github.com> On Tue, 20 Sep 2022 21:05:31 GMT, Brian Burkhalter wrote: >> Maxim Kartashev has updated the pull request incrementally with one additional commit since the last revision: >> >> Removed the change unnecessary without recursive watch > > src/java.base/macosx/native/libnio/fs/MacOSXWatchService.c line 199: > >> 197: >> 198: FSEventStreamStop(streamRef); // Unregister with the FS Events service. >> 199: // No more callbacks from this stream. > > Does `FSEventStreamUnscheduleFromRunLoop()` need to be called here? I think not. The [documentation](https://developer.apple.com/documentation/coreservices/1446990-fseventstreaminvalidate?language=objc) for `FSEventStreamInvalidate()` that gets called immediately after this line states the following: > Invalidates the stream, like CFRunLoopSourceInvalidate() does for a CFRunLoopSourceRef. It will be unscheduled from any runloops or dispatch queues upon which it had been scheduled. So calling `FSEventStreamUnscheduleFromRunLoop()` seems superfluous; its semantics is described as > ... removes the stream from the specified run loop, like CFRunLoopRemoveSource() does for a CFRunLoopSourceRef. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From mkartashev at openjdk.org Thu Sep 22 08:18:07 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Thu, 22 Sep 2022 08:18:07 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v4] In-Reply-To: References: Message-ID: > This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. > > Some things of note: > * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. > * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). > * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). > * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). > * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. > > This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. > > ### Testing > > * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). > * Also verified that new tests pass on Linux and Windows. > * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. Maxim Kartashev has updated the pull request incrementally with one additional commit since the last revision: Minor changes based on review comments ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10140/files - new: https://git.openjdk.org/jdk/pull/10140/files/94421827..46cf5c54 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=03 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=02-03 Stats: 13 lines in 2 files changed: 4 ins; 5 del; 4 mod Patch: https://git.openjdk.org/jdk/pull/10140.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10140/head:pull/10140 PR: https://git.openjdk.org/jdk/pull/10140 From mkartashev at openjdk.org Thu Sep 22 08:57:21 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Thu, 22 Sep 2022 08:57:21 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v5] In-Reply-To: References: Message-ID: > This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. > > Some things of note: > * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. > * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). > * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). > * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). > * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. > > This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. > > ### Testing > > * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). > * Also verified that new tests pass on Linux and Windows. > * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. Maxim Kartashev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains ten commits: - Merge master - Minor changes based on review comments - Removed the change unnecessary without recursive watch - Removed support for recursive watch - Reformatted native code and removed jlong and pointer casts - Removed the dependency on provider(Path) - More robust way of obtaining run loop reference - CFRunLoopThread to use InnocuousThread - 8293067: (fs) Implement WatchService using system library (macOS) ------------- Changes: https://git.openjdk.org/jdk/pull/10140/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=04 Stats: 1157 lines in 9 files changed: 1149 ins; 3 del; 5 mod Patch: https://git.openjdk.org/jdk/pull/10140.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10140/head:pull/10140 PR: https://git.openjdk.org/jdk/pull/10140 From jpai at openjdk.org Thu Sep 22 09:20:26 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Thu, 22 Sep 2022 09:20:26 GMT Subject: RFR: 8294193: Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory [v2] In-Reply-To: References: Message-ID: <7O-GzppAT4q9mGDkH76C0wCXCp320SPM-QoVqRMjnOU=.35547beb-67d1-411d-b015-02dbcb01e5c0@github.com> > Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8294193? > > As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. > tier1, tier2 and tier3 tests passed with this change. Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: move Files.createDirectories testing from Misc.java to CreateDirectories.java ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10383/files - new: https://git.openjdk.org/jdk/pull/10383/files/daf6514a..e918d33c Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10383&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10383&range=00-01 Stats: 87 lines in 2 files changed: 45 ins; 34 del; 8 mod Patch: https://git.openjdk.org/jdk/pull/10383.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10383/head:pull/10383 PR: https://git.openjdk.org/jdk/pull/10383 From jpai at openjdk.org Thu Sep 22 09:20:27 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Thu, 22 Sep 2022 09:20:27 GMT Subject: RFR: 8294193: Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory In-Reply-To: References: Message-ID: <4X7E3S76a19rhoUd9VfogjQFjjdipMNAsbJKYc0cmX4=.b41d480f-7bff-4fec-a2e1-db29198e0425@github.com> On Thu, 22 Sep 2022 06:34:53 GMT, Jaikiran Pai wrote: > Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8294193? > > As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. > tier1, tier2 and tier3 tests passed with this change. Hello Alan, > I think create a new issue for Files.createDirectories issue or else move the existing issue from hotspot/runtime and change the description. I've now created a separate JBS and this PR has been updated to link to that JBS. > For the test then I think it will mean moving the existing tests from Misc.java to the new source file. That will fit with the description that you propose for the new test. I've updated the PR to move the `Files.createDirectories` testing from `Misc.java` to this new test class. I have done some trivial cosmetic/non-functional changes to this moved test (things like printing an assert message if/when any assertion fails in that test). > I plan to review this change but not today because it require digging into some security issues. Please take your time. ------------- PR: https://git.openjdk.org/jdk/pull/10383 From bpb at openjdk.org Thu Sep 22 16:03:03 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 22 Sep 2022 16:03:03 GMT Subject: RFR: 8293331: Refactor FileDispatcherImpl into operating system-specific components [v8] In-Reply-To: <1n-gpAD-GQPUYUIFI7IPoLPo1jQQPk6orRVaxbVV88Q=.6123e39b-1051-44cb-a5af-edd38ff0ecdc@github.com> References: <1n-gpAD-GQPUYUIFI7IPoLPo1jQQPk6orRVaxbVV88Q=.6123e39b-1051-44cb-a5af-edd38ff0ecdc@github.com> Message-ID: On Thu, 22 Sep 2022 06:00:38 GMT, Vyom Tewari wrote: >> Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: >> >> 8293331: Add missing init() call in UnixFileDispatcherImpl static initializer > > src/java.base/linux/classes/sun/nio/ch/FileDispatcherImpl.java line 51: > >> 49: // -- Native methods -- >> 50: >> 51: static native long transferTo0(FileDescriptor src, long position, > > can we make these native methods(transferTo0, transferFrom0 etc) to private access specifier, these methods are not called from outside ?. This comment applies to a lot of code. See `UnixFileDispatcherImpl` for example. I prefer not to change it at this time. ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Thu Sep 22 16:03:05 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 22 Sep 2022 16:03:05 GMT Subject: Integrated: 8293331: Refactor FileDispatcherImpl into operating system-specific components In-Reply-To: References: Message-ID: <-HMc8KBYpjEKhbBg_XOMTuWfmXEo7olujQYAlMlFY8E=.2a9d34d2-b7cc-4848-8ddc-0894e647d13c@github.com> On Sat, 3 Sep 2022 00:59:07 GMT, Brian Burkhalter wrote: > Break up `sun.nio.ch.FileChannelImpl` in a manner similar but not identical to the refactoring of `sun.nio.fs.UnixFileSystem` performed in #10093. This pull request has now been integrated. Changeset: 48cc1560 Author: Brian Burkhalter URL: https://git.openjdk.org/jdk/commit/48cc15602b62e81bb179ca9570a1e7d8bbf4d6df Stats: 2657 lines in 16 files changed: 1443 ins; 1188 del; 26 mod 8293331: Refactor FileDispatcherImpl into operating system-specific components Reviewed-by: alanb ------------- PR: https://git.openjdk.org/jdk/pull/10154 From bpb at openjdk.org Thu Sep 22 18:13:24 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 22 Sep 2022 18:13:24 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v3] In-Reply-To: <4tbbedLVW5VADXjUbXB7CssAy6d4muCAaIPxIosfF9k=.5c003921-08d3-424a-a964-95522f415472@github.com> References: <4tbbedLVW5VADXjUbXB7CssAy6d4muCAaIPxIosfF9k=.5c003921-08d3-424a-a964-95522f415472@github.com> Message-ID: <_bln4R8XcYWq5TgWZnGscVWEs1c6wbyAmWmfWSaJPdE=.efec752f-8b74-4e93-a0c8-dd0a3461d566@github.com> On Thu, 22 Sep 2022 07:06:46 GMT, Maxim Kartashev wrote: >> src/java.base/macosx/classes/sun/nio/fs/MacOSXWatchService.java line 399: >> >>> 397: } >>> 398: >>> 399: final int flags = unsafe.getInt(eventFlagsPtr); >> >> This looks like the only use of `Unsafe`. Could the objective be accomplished without using it, e.g., changing `long eventFlagsPtr` to a `long[]` or, maybe even simpler, a `boolean[]`? > > Earlier on the mailing list @AlanBateman wrote "It would help the review if the native code is just simple wrappers about the FSEvents as we'll likely replace a lot of the JNI code in this area in time." > I understood that as a suggestion to keep the native code as simple as possible and, conversely, do as much as possible on the Java side. Reading native memory here was one of the decisions that were based on that understanding. > I may have, of course, read too much into it, so I'm not opposed to implementing your suggestion. I would like, however, to have a confirmation that this direction of the changes is OK with @AlanBateman also. Your understanding is correct, but sometimes precisely mapping the native API can result in more complication than needed. While this might not be a strictly analogous example, consider the BSD system call `setattrlist(2)`. We only need to use it at present to set file times, so this call was exposed at the Java level with a simpler set of parameters. See [sun.nio.fs.BsdNativeDispatcher::setattrlist](https://github.com/openjdk/jdk/blob/5285035ed9bb43a40108e4d046e0de317730f193/src/java.base/macosx/classes/sun/nio/fs/BsdNativeDispatcher.java#L84 ). I?ll leave it to @AlanBateman to express his own opinion on this. >> src/java.base/macosx/native/libnio/fs/MacOSXWatchService.c line 73: >> >>> 71: (*env)->CallVoidMethod(env, watchService, callbackMID, >>> 72: streamRef, javaEventPathsArray, eventFlags); >>> 73: } >> >> Is there a way to notify of events without an upcall? Maybe add the necessary values to a queue of some sort observed by the Java layer? > > I'm not sure I know how to synchronize native code with Java without any upcall to the Java side; but maybe you are suggesting something as simple as a `notify()` call in place of the current callback with many arguments?. > > And also this goes against my understanding that the native part has to be kept as simple and as straightforward as possible. Having it maintain a queue that has to be synchronized with Java will certainly complicate the native part. I was thinking of maybe something like a [BlockingQueue](https://docs.oracle.com/en/java/javase/19/docs/api/java.base/java/util/concurrent/BlockingQueue.html) field on `sun.nio.fs.MacOSXWatchService`, but I am not sure whether this is any more correct. If the callback is retained, it might be worth considering using [JNU_CallMethodByName](https://github.com/openjdk/jdk/blob/5285035ed9bb43a40108e4d046e0de317730f193/src/java.base/share/native/libjava/jni_util.h#L134). >> src/java.base/macosx/native/libnio/fs/MacOSXWatchService.c line 89: >> >>> 87: { >>> 88: JNIEnv *env = (JNIEnv *) JNU_GetEnv(jvm, JNI_VERSION_1_2); >>> 89: if (!env) { // Shouldn't happen as run loop starts from Java code >> >> Is this check, hence the `jvm` extern, really necessary? > > The check isn't strictly necessary, so I'm goign to drop it (there's a JNI version check in `nio_util.c`), but the `jvm` extern seems necessary because that's how `JNIEnv` is obtained and the latter is necessary to communicate the events back to the Java side. I don?t know whether it?s possible to get rid of the `jvm` extern, i.e. , whether there?s another way to get the `JNIEnv`. >> src/java.base/macosx/native/libnio/fs/MacOSXWatchService.c line 199: >> >>> 197: >>> 198: FSEventStreamStop(streamRef); // Unregister with the FS Events service. >>> 199: // No more callbacks from this stream. >> >> Does `FSEventStreamUnscheduleFromRunLoop()` need to be called here? > > I think not. The [documentation](https://developer.apple.com/documentation/coreservices/1446990-fseventstreaminvalidate?language=objc) for `FSEventStreamInvalidate()` that gets called immediately after this line states the following: >> Invalidates the stream, like CFRunLoopSourceInvalidate() does for a CFRunLoopSourceRef. It will be unscheduled from any runloops or dispatch queues upon which it had been scheduled. > > So calling `FSEventStreamUnscheduleFromRunLoop()` seems superfluous; its semantics is described as >> ... removes the stream from the specified run loop, like CFRunLoopRemoveSource() does for a CFRunLoopSourceRef. That sounds correct. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From manc at google.com Thu Sep 22 23:26:38 2022 From: manc at google.com (Man Cao) Date: Thu, 22 Sep 2022 16:26:38 -0700 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? Message-ID: Hi all, I recently opened an RFE (https://bugs.openjdk.org/browse/JDK-8294052) to make JVM flags such as -XX:+HeapDumpOnOutOfMemoryError and -XX:+ExitOnOutOfMemoryError work for OutOfMemoryErrors from filling up the NIO direct memory. Supporting HeapDumpOnOutOfMemoryError would greatly help users debug OOMEs from NIO direct memory. In our case, the OOMEs from direct memory are infrequent and unpredictable, and it is quite infeasible to manually trigger a heap dump just before those OOMEs happen. However, [~dholmes] seems a bit wary of supporting those JVM flags for NIO direct memory, as those JVM flags are currently for OOMEs thrown from the JVM, and for OOMEs about Java heap and metaspace, which are both managed by the JVM. Do you think it is a good idea to support those JVM flags for OOMEs from NIO direct memory? Are there any concerns that we shouldn't support them? -Man -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Fri Sep 23 04:05:31 2022 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 23 Sep 2022 06:05:31 +0200 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? In-Reply-To: References: Message-ID: My opinion? Yes, these flags should work for all kind of OOMEs that indicate a real resource exhaustion, regardless of its point of origin. Our users don't care, they just want their heap dumps to work reliably. But its an unsolved point of contention. We have tried to get this into the OpenJDK repeatedly, but failed to convince Oracle. Therefore we gave up and changed behavior in our downstream JVMs: https://github.com/SAP/SapMachine/wiki/Handling-of-OnOutOfMemoryError-switches-in-the-SapMachine which is always regrettable - tiny steps toward fragmentation of the JVM landscape. But we needed to make our users happy. Cheers, Thomas On Fri, Sep 23, 2022 at 1:27 AM Man Cao wrote: > Hi all, > > I recently opened an RFE (https://bugs.openjdk.org/browse/JDK-8294052) to > make JVM flags such as -XX:+HeapDumpOnOutOfMemoryError and > -XX:+ExitOnOutOfMemoryError work for OutOfMemoryErrors from filling up the > NIO direct memory. > Supporting HeapDumpOnOutOfMemoryError would greatly help users debug OOMEs > from NIO direct memory. In our case, the OOMEs from direct memory are > infrequent and unpredictable, and it is quite infeasible to > manually trigger a heap dump just before those OOMEs happen. > > However, [~dholmes] seems a bit wary of supporting those JVM flags for NIO > direct memory, as those JVM flags are currently for OOMEs thrown from the > JVM, and for OOMEs about Java heap and metaspace, which are both managed by > the JVM. > > Do you think it is a good idea to support those JVM flags for OOMEs from > NIO direct memory? Are there any concerns that we shouldn't support them? > > -Man > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Sep 23 08:20:45 2022 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 23 Sep 2022 09:20:45 +0100 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? In-Reply-To: References: Message-ID: <7e024658-20bc-fe9a-38ef-f668ea5a4e5e@oracle.com> On 23/09/2022 00:26, Man Cao wrote: > Hi all, > > I recently opened an RFE > (https://bugs.openjdk.org/browse/JDK-8294052)?to make JVM flags such > as -XX:+HeapDumpOnOutOfMemoryError and -XX:+ExitOnOutOfMemoryError > work for OutOfMemoryErrors from filling up the NIO direct memory. > Supporting HeapDumpOnOutOfMemoryError would greatly help users debug > OOMEs from NIO direct memory. In our case, the OOMEs from direct > memory are infrequent and unpredictable, and it is quite infeasible to > manually?trigger a heap dump just before those OOMEs happen. > > However, [~dholmes] seems a bit wary of supporting those JVM flags for > NIO direct memory, as those JVM flags are currently for OOMEs thrown > from the JVM, and for OOMEs about Java heap and metaspace, which are > both managed by the JVM. > > Do you think it is a good idea to support those JVM flags for OOMEs > from NIO direct memory? Are there any concerns that we shouldn't > support them? > There are many places in the library code that throw OOME. One common case in APIs is where some parameters or conditions require allocating an array larger than 2Gb. That's not possible of course so some exception needs to be thrown and it is usually OOME (this is specified in some cases, not in others). The error can be confusing as there isn't really an out of memory (java or native heap). These cases are probably not good candidates to trigger a heap dump or cause the VM to exit. There are other cases (200+) where malloc or some other allocation in native code fails. These throw OOME (via JNU_ThrowOutOfMemoryError as you've probably found). There have been debates over the years as to whether OOME is the right error to throw as seeing this exception in a log can can make some some people think they need to increase -Xmx when the issue is somewhere else. If the native heap is completely exhausted then there's a good chance the VM will crash or exit quickly. Should a heap dump or these OOME trigger the VM to exit? Maybe in some cases where native resources are being kept alive but some object that would be expected to be GC'ed. I realise the focus of JDK-8294052 has narrowed since it was created but I think it would be fair to say that it would probably require going through them case by case to see if it make sense. I think there is a case for triggering a heap dump or doing the other OOME actions when a direct buffer can't be allocated. Direct memory isn't managed by the VM but we know that hitting the limit means there are objects keeping the direct buffers alive and there is tuning (via a XX option) that may be required for some applications, and maybe a memory leak in others. Related is that we added BufferPoolMXBean in Java 7 so that direct memory could be monitored by JMX tooling on the same level as memory pools managed by the VM. -Alan From thomas.stuefe at gmail.com Fri Sep 23 09:14:36 2022 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Fri, 23 Sep 2022 11:14:36 +0200 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? In-Reply-To: <7e024658-20bc-fe9a-38ef-f668ea5a4e5e@oracle.com> References: <7e024658-20bc-fe9a-38ef-f668ea5a4e5e@oracle.com> Message-ID: Hi Alan, On Fri, Sep 23, 2022 at 10:21 AM Alan Bateman wrote: > On 23/09/2022 00:26, Man Cao wrote: > > Hi all, > > > > I recently opened an RFE > > (https://bugs.openjdk.org/browse/JDK-8294052) to make JVM flags such > > as -XX:+HeapDumpOnOutOfMemoryError and -XX:+ExitOnOutOfMemoryError > > work for OutOfMemoryErrors from filling up the NIO direct memory. > > Supporting HeapDumpOnOutOfMemoryError would greatly help users debug > > OOMEs from NIO direct memory. In our case, the OOMEs from direct > > memory are infrequent and unpredictable, and it is quite infeasible to > > manually trigger a heap dump just before those OOMEs happen. > > > > However, [~dholmes] seems a bit wary of supporting those JVM flags for > > NIO direct memory, as those JVM flags are currently for OOMEs thrown > > from the JVM, and for OOMEs about Java heap and metaspace, which are > > both managed by the JVM. > > > > Do you think it is a good idea to support those JVM flags for OOMEs > > from NIO direct memory? Are there any concerns that we shouldn't > > support them? > > > > There are many places in the library code that throw OOME. > > One common case in APIs is where some parameters or conditions require > allocating an array larger than 2Gb. That's not possible of course so > some exception needs to be thrown and it is usually OOME (this is > specified in some cases, not in others). The error can be confusing as > there isn't really an out of memory (java or native heap). These cases > are probably not good candidates to trigger a heap dump or cause the VM > to exit. > > There are other cases (200+) where malloc or some other allocation in > native code fails. These throw OOME (via JNU_ThrowOutOfMemoryError as > you've probably found). There have been debates over the years as to > whether OOME is the right error to throw as seeing this exception in a > log can can make some some people think they need to increase -Xmx when > the issue is somewhere else. If the native heap is completely exhausted > then there's a good chance the VM will crash or exit quickly. Should a > heap dump or these OOME trigger the VM to exit? Maybe in some cases > where native resources are being kept alive but some object that would > be expected to be GC'ed. I realise the focus of JDK-8294052 has narrowed > since it was created but I think it would be fair to say that it would > probably require going through them case by case to see if it make sense. > > I think there is a case for triggering a heap dump or doing the other > OOME actions when a direct buffer can't be allocated. Direct memory > isn't managed by the VM but we know that hitting the limit means there > are objects keeping the direct buffers alive and there is tuning (via a > XX option) that may be required for some applications, and maybe a > memory leak in others. Related is that we added BufferPoolMXBean in Java > 7 so that direct memory could be monitored by JMX tooling on the same > level as memory pools managed by the VM. > > -Alan > Thank you for the detailed explanation. Now I remember again how complex this issue was. For us, even if a heap dump can be of limited use when analyzing e.g. thread start errors, the other OOM actions can still be quite useful. Our customers often just wish for a way to quickly exit the VM, to be able to restart it. Or to crash it quickly, to get a hs-err file with information which are often more geared toward analyzing native OOM situations (e.g. rlimits, NMT, metaspace report, or the process' pmap...). We found that customers are often confused that ..OnOutOfMemory switches don't work for all cases of OOMEs. They come up with different mechanics, e.g. monitoring JVMTI ResourceExhausted. Which also does not fire for every resource exhaustion, e.g. if a native OOM occurs in the compiler. Out of desperation, we added a facility do our downstream VM to react on impending resource exhaustion with, among other things, the ability to fire up jcmds automatically. Before the process gets OOM-killed and vanishes without a trace ( https://github.com/SAP/SapMachine/wiki/SapMachine-High-Memory-Reports) - similar to SIGDANGER on some proprietary Unices. Cheers, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From alanb at openjdk.org Fri Sep 23 14:30:15 2022 From: alanb at openjdk.org (Alan Bateman) Date: Fri, 23 Sep 2022 14:30:15 GMT Subject: RFR: 8294193: Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory [v2] In-Reply-To: <7O-GzppAT4q9mGDkH76C0wCXCp320SPM-QoVqRMjnOU=.35547beb-67d1-411d-b015-02dbcb01e5c0@github.com> References: <7O-GzppAT4q9mGDkH76C0wCXCp320SPM-QoVqRMjnOU=.35547beb-67d1-411d-b015-02dbcb01e5c0@github.com> Message-ID: On Thu, 22 Sep 2022 09:20:26 GMT, Jaikiran Pai wrote: >> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8294193? >> >> As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. >> tier1, tier2 and tier3 tests passed with this change. > > Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: > > move Files.createDirectories testing from Misc.java to CreateDirectories.java test/jdk/java/nio/file/Files/CreateDirectories.java line 44: > 42: /** > 43: * Creates a symlink which points to a directory that exists. Then calls Files.createDirectories > 44: * method passing it the path to the symlink and verifies that no exception gets thrown I think you can simplify the wording with something like "Test Files.createDirectories symbolic file with an existing directory." test/jdk/java/nio/file/Files/CreateDirectories.java line 49: > 47: public void testSymlinkDir() throws Exception { > 48: // create a temp dir as the "root" in which we will run our tests. > 49: final Path startingDir = Files.createTempDirectory(Path.of("."), "8293792"); The tests in this area use TestUtil.createTemporaryDirectory() so it's probably best to keep it consistent if you can. Also can you rename "startingDir" to "top" as that will make the test much easier to read. test/jdk/java/nio/file/Files/CreateDirectories.java line 56: > 54: } > 55: System.out.println("Running tests under directory " + startingDir.toAbsolutePath()); > 56: final Path fooDir = Files.createDirectory(Path.of(startingDir.toString(), "foo")); You can simplify this to `Files.createDirectory(top.resolve("foo"))`, same for barDir, and symlink. test/jdk/java/nio/file/Files/CreateDirectories.java line 100: > 98: Path subdir = tmpdir.resolve("a"); > 99: Files.createDirectories(subdir); > 100: assertTrue(Files.exists(subdir), subdir + " was expected to exist, but didn't"); I realise this is moved from Misc.java but if you replace the "exists" with "isDirectory" then it improve the testing. ------------- PR: https://git.openjdk.org/jdk/pull/10383 From manc at google.com Fri Sep 23 22:24:21 2022 From: manc at google.com (Man Cao) Date: Fri, 23 Sep 2022 15:24:21 -0700 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? In-Reply-To: References: <7e024658-20bc-fe9a-38ef-f668ea5a4e5e@oracle.com> Message-ID: Thank you both for the feedback. > I realise the focus of JDK-8294052 has narrowed > since it was created but I think it would be fair to say that it would > probably require going through them case by case to see if it make sense. Yes. I completely agree that the OOMEs should be analyzed case by case. Also completely agree that the OOMEs for large array allocations do not need to handle these JVM flags. > I think there is a case for triggering a heap dump or doing the other > OOME actions when a direct buffer can't be allocated. > ... It sounds like I could proceed sending the RFR to support these JVM flags for direct memory OOME? In our case, the application sets -XX:MaxDirectMemorySize and it ran out of this limit. While bumping up MaxDirectMemorySize could solve the problem, we'd like to understand where the increase comes from. We already monitor direct memory usage via BufferPoolMXBean, but it is not sufficient to dig into the root cause. We really heap dumps to understand the root cause. -Man On Fri, Sep 23, 2022 at 2:14 AM Thomas St?fe wrote: > Hi Alan, > > On Fri, Sep 23, 2022 at 10:21 AM Alan Bateman > wrote: > >> On 23/09/2022 00:26, Man Cao wrote: >> > Hi all, >> > >> > I recently opened an RFE >> > (https://bugs.openjdk.org/browse/JDK-8294052) to make JVM flags such >> > as -XX:+HeapDumpOnOutOfMemoryError and -XX:+ExitOnOutOfMemoryError >> > work for OutOfMemoryErrors from filling up the NIO direct memory. >> > Supporting HeapDumpOnOutOfMemoryError would greatly help users debug >> > OOMEs from NIO direct memory. In our case, the OOMEs from direct >> > memory are infrequent and unpredictable, and it is quite infeasible to >> > manually trigger a heap dump just before those OOMEs happen. >> > >> > However, [~dholmes] seems a bit wary of supporting those JVM flags for >> > NIO direct memory, as those JVM flags are currently for OOMEs thrown >> > from the JVM, and for OOMEs about Java heap and metaspace, which are >> > both managed by the JVM. >> > >> > Do you think it is a good idea to support those JVM flags for OOMEs >> > from NIO direct memory? Are there any concerns that we shouldn't >> > support them? >> > >> >> There are many places in the library code that throw OOME. >> >> One common case in APIs is where some parameters or conditions require >> allocating an array larger than 2Gb. That's not possible of course so >> some exception needs to be thrown and it is usually OOME (this is >> specified in some cases, not in others). The error can be confusing as >> there isn't really an out of memory (java or native heap). These cases >> are probably not good candidates to trigger a heap dump or cause the VM >> to exit. >> >> There are other cases (200+) where malloc or some other allocation in >> native code fails. These throw OOME (via JNU_ThrowOutOfMemoryError as >> you've probably found). There have been debates over the years as to >> whether OOME is the right error to throw as seeing this exception in a >> log can can make some some people think they need to increase -Xmx when >> the issue is somewhere else. If the native heap is completely exhausted >> then there's a good chance the VM will crash or exit quickly. Should a >> heap dump or these OOME trigger the VM to exit? Maybe in some cases >> where native resources are being kept alive but some object that would >> be expected to be GC'ed. I realise the focus of JDK-8294052 has narrowed >> since it was created but I think it would be fair to say that it would >> probably require going through them case by case to see if it make sense. >> >> I think there is a case for triggering a heap dump or doing the other >> OOME actions when a direct buffer can't be allocated. Direct memory >> isn't managed by the VM but we know that hitting the limit means there >> are objects keeping the direct buffers alive and there is tuning (via a >> XX option) that may be required for some applications, and maybe a >> memory leak in others. Related is that we added BufferPoolMXBean in Java >> 7 so that direct memory could be monitored by JMX tooling on the same >> level as memory pools managed by the VM. >> >> -Alan >> > > Thank you for the detailed explanation. Now I remember again how complex > this issue was. > > For us, even if a heap dump can be of limited use when analyzing e.g. > thread start errors, the other OOM actions can still be quite useful. Our > customers often just wish for a way to quickly exit the VM, to be able to > restart it. Or to crash it quickly, to get a hs-err file with information > which are often more geared toward analyzing native OOM situations (e.g. > rlimits, NMT, metaspace report, or the process' pmap...). > > We found that customers are often confused that ..OnOutOfMemory switches > don't work for all cases of OOMEs. They come up with different mechanics, > e.g. monitoring JVMTI ResourceExhausted. Which also does not fire for every > resource exhaustion, e.g. if a native OOM occurs in the compiler. > > Out of desperation, we added a facility do our downstream VM to react on > impending resource exhaustion with, among other things, the ability to fire > up jcmds automatically. Before the process gets OOM-killed and vanishes > without a trace ( > https://github.com/SAP/SapMachine/wiki/SapMachine-High-Memory-Reports) - > similar to SIGDANGER on some proprietary Unices. > > Cheers, Thomas > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpai at openjdk.org Sat Sep 24 06:11:21 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Sat, 24 Sep 2022 06:11:21 GMT Subject: RFR: 8294193: Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory [v3] In-Reply-To: References: Message-ID: > Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8294193? > > As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. > tier1, tier2 and tier3 tests passed with this change. Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: review comments for test ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10383/files - new: https://git.openjdk.org/jdk/pull/10383/files/e918d33c..350b3f87 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10383&range=02 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10383&range=01-02 Stats: 14 lines in 1 file changed: 2 ins; 1 del; 11 mod Patch: https://git.openjdk.org/jdk/pull/10383.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10383/head:pull/10383 PR: https://git.openjdk.org/jdk/pull/10383 From jpai at openjdk.org Sat Sep 24 06:11:23 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Sat, 24 Sep 2022 06:11:23 GMT Subject: RFR: 8294193: Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory [v2] In-Reply-To: <7O-GzppAT4q9mGDkH76C0wCXCp320SPM-QoVqRMjnOU=.35547beb-67d1-411d-b015-02dbcb01e5c0@github.com> References: <7O-GzppAT4q9mGDkH76C0wCXCp320SPM-QoVqRMjnOU=.35547beb-67d1-411d-b015-02dbcb01e5c0@github.com> Message-ID: On Thu, 22 Sep 2022 09:20:26 GMT, Jaikiran Pai wrote: >> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8294193? >> >> As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. >> tier1, tier2 and tier3 tests passed with this change. > > Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: > > move Files.createDirectories testing from Misc.java to CreateDirectories.java Hello Alan, I've updated the PR to implement your suggestions in the test. The test continues to pass. ------------- PR: https://git.openjdk.org/jdk/pull/10383 From alanb at openjdk.org Sat Sep 24 08:47:16 2022 From: alanb at openjdk.org (Alan Bateman) Date: Sat, 24 Sep 2022 08:47:16 GMT Subject: RFR: 8294193: Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory [v3] In-Reply-To: References: Message-ID: On Sat, 24 Sep 2022 06:11:21 GMT, Jaikiran Pai wrote: >> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8294193? >> >> As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. >> tier1, tier2 and tier3 tests passed with this change. > > Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: > > review comments for test Thanks for the updates and for taking this issue. ------------- Marked as reviewed by alanb (Reviewer). PR: https://git.openjdk.org/jdk/pull/10383 From Alan.Bateman at oracle.com Sun Sep 25 08:10:40 2022 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 25 Sep 2022 09:10:40 +0100 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? In-Reply-To: References: <7e024658-20bc-fe9a-38ef-f668ea5a4e5e@oracle.com> Message-ID: <30ea76f7-cdfe-b357-8427-0fd0627dd3fa@oracle.com> On 23/09/2022 23:24, Man Cao wrote: > : > > Yes. I completely agree that the OOMEs should be analyzed case by case. > Also completely agree that the OOMEs for large array allocations do > not need to handle these JVM flags. I think OOME has been over used and if we could go back in time then maybe things would be different and OOME would not be thrown for some of these cases. > : > > It sounds like I could proceed sending the RFR to support these JVM > flags for direct memory OOME? > In our case, the application sets -XX:MaxDirectMemorySize and it ran > out of this limit. > While bumping up MaxDirectMemorySize could solve the problem, we'd > like to understand where the increase comes from. We already monitor > direct memory usage via BufferPoolMXBean, but it is not sufficient to > dig into the root cause. We really heap dumps to understand the root > cause. > I don't have any objection to this. I only mentioned BufferPoolMXBean to show that memory for direct and mapped buffers is already exposed for monitoring purposes. Also direct buffers are a clear case where the memory is kept alive by live objects and at the same time have a runtime imposed limit. With the foreign memory API (Project Panama) it means there will also be memory segments that are part of lifecycles managed by memory sessions - in time, these will also be resources that may need to be monitored and maybe trigger heap dumps / OOME actions too. -Alan From mernst at openjdk.org Sun Sep 25 17:04:38 2022 From: mernst at openjdk.org (Michael Ernst) Date: Sun, 25 Sep 2022 17:04:38 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: Message-ID: <_zVJ429VO5Y3YZu0eDp_RD4BZRgEOBdDIIBuGf9vH6I=.b17025c7-1e0b-4a1b-a756-3a971b76bd03@github.com> On Thu, 25 Aug 2022 15:35:53 GMT, Michael Ernst wrote: > 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni > If you already are an OpenJDK [Author](https://openjdk.java.net/bylaws#author), [Committer](https://openjdk.java.net/bylaws#committer) or [Reviewer](https://openjdk.java.net/bylaws#reviewer), please click [here](https://bugs.openjdk.java.net/secure/CreateIssue.jspa?pid=11300&issuetype=1) to open a new issue so that we can record that fact. Please use "Add GitHub user mernst" as summary for the issue. Done: https://bugs.openjdk.org/browse/SKARA-1566 ------------- PR: https://git.openjdk.org/jdk/pull/10029 From mernst at openjdk.org Sun Sep 25 17:04:38 2022 From: mernst at openjdk.org (Michael Ernst) Date: Sun, 25 Sep 2022 17:04:38 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni Message-ID: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni ------------- Commit messages: - Remove file that was removed upstream - Fix inconsistency in capitalization - Undo change in zlip - Fix typos Changes: https://git.openjdk.org/jdk/pull/10029/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10029&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8294321 Stats: 317 lines in 108 files changed: 0 ins; 195 del; 122 mod Patch: https://git.openjdk.org/jdk/pull/10029.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10029/head:pull/10029 PR: https://git.openjdk.org/jdk/pull/10029 From jpai at openjdk.org Sun Sep 25 17:04:38 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Sun, 25 Sep 2022 17:04:38 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: <_zVJ429VO5Y3YZu0eDp_RD4BZRgEOBdDIIBuGf9vH6I=.b17025c7-1e0b-4a1b-a756-3a971b76bd03@github.com> References: <_zVJ429VO5Y3YZu0eDp_RD4BZRgEOBdDIIBuGf9vH6I=.b17025c7-1e0b-4a1b-a756-3a971b76bd03@github.com> Message-ID: On Thu, 25 Aug 2022 19:55:47 GMT, Michael Ernst wrote: >> 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni > >> If you already are an OpenJDK [Author](https://openjdk.java.net/bylaws#author), [Committer](https://openjdk.java.net/bylaws#committer) or [Reviewer](https://openjdk.java.net/bylaws#reviewer), please click [here](https://bugs.openjdk.java.net/secure/CreateIssue.jspa?pid=11300&issuetype=1) to open a new issue so that we can record that fact. Please use "Add GitHub user mernst" as summary for the issue. > > Done: https://bugs.openjdk.org/browse/SKARA-1566 Hello Michael @mernst, these changes are mostly fine. However, they are spread across files which reside in different areas of the JDK. For changes like these, we usually split up the changes into different PRs to help reviewers of specific areas focus on the individual PRs. Would you be willing to split this PR into separate ones. I would recommend starting with one PR which includes changes that are part of `test/jdk/java`, `test/jdk/jdk` and `test/jdk/jni`. If you can come up with that PR, then I'll go ahead and create a JBS issue for it and you can then link the PR against that issue. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From mernst at openjdk.org Sun Sep 25 17:04:38 2022 From: mernst at openjdk.org (Michael Ernst) Date: Sun, 25 Sep 2022 17:04:38 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: <_zVJ429VO5Y3YZu0eDp_RD4BZRgEOBdDIIBuGf9vH6I=.b17025c7-1e0b-4a1b-a756-3a971b76bd03@github.com> Message-ID: On Mon, 5 Sep 2022 06:32:44 GMT, Jaikiran Pai wrote: > these changes are mostly fine. Can you be specific about the exact problems that you noticed that prevented you from saying "these changes are fine"? It doesn't seem useful to make a pull request with known deficiencies. > we usually split up the changes into different PRs to help reviewers of specific areas focus on the individual PRs. Feel free to split up the pull request, if you feel that specific expertise in specific parts of the JDK is necessary to review these changes. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From duke at openjdk.org Sun Sep 25 17:04:39 2022 From: duke at openjdk.org (SWinxy) Date: Sun, 25 Sep 2022 17:04:39 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: <_zVJ429VO5Y3YZu0eDp_RD4BZRgEOBdDIIBuGf9vH6I=.b17025c7-1e0b-4a1b-a756-3a971b76bd03@github.com> Message-ID: On Mon, 5 Sep 2022 14:54:50 GMT, Michael Ernst wrote: > Can you be specific about the exact problems that you noticed that prevented you from saying "these changes are fine"? These changes are fine. I don't see an instance where the duplicated words would mean a change in the specification. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From jpai at openjdk.org Sun Sep 25 17:04:39 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Sun, 25 Sep 2022 17:04:39 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: Message-ID: On Thu, 25 Aug 2022 15:35:53 GMT, Michael Ernst wrote: > 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni Hello Michael, > > these changes are mostly fine. > > Can you be specific about the exact problems that you noticed that prevented you from saying "these changes are fine"? It doesn't seem useful to make a pull request with known deficiencies. I wasn't implying this PR had deficiencies. What I meant was, to be able to have this reviewed, in addition to code changes this will also need a issue to be created and linked to this PR (and the PR title changed to the form `: `). I have now created https://bugs.openjdk.org/browse/JDK-8294321. Splitting these changes into smaller PRs instead of one big one that touches several different areas of the JDK and 108 files, will help reviewers. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From mernst at openjdk.org Sun Sep 25 17:04:39 2022 From: mernst at openjdk.org (Michael Ernst) Date: Sun, 25 Sep 2022 17:04:39 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: Message-ID: On Sat, 24 Sep 2022 10:41:05 GMT, Jaikiran Pai wrote: > Splitting these changes into smaller PRs instead of one big one that touches several different areas of the JDK and 108 files, will help reviewers. OK. I'll repeat my comment from before: > Feel free to split up the pull request, if you feel that specific expertise in specific parts of the JDK is necessary to review these changes. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From mernst at openjdk.org Sun Sep 25 17:04:39 2022 From: mernst at openjdk.org (Michael Ernst) Date: Sun, 25 Sep 2022 17:04:39 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: Message-ID: On Sun, 25 Sep 2022 16:51:27 GMT, SWinxy wrote: > Oh no what happened Oops! Thanks for pointing out the goof. It seems to be fixed now. It has now been a month since I opened the pull request, and I have responded to each substantive comment promptly. It would be great to merge this pull request to avoid the need for more merges. > src/java.xml/share/classes/com/sun/org/apache/xpath/internal/NodeSetDTM.java line 1053: > >> 1051: * fetch will take place at index 1. >> 1052: * >> 1053: * @return The current position index. > > Note that there is an inconsistent change here, albeit insignificant. In `NodeSet`, the capital 'The' is removed after the `@return`, while the lowercase 'the' is removed here. Thanks for pointing out the inconsistency. I have fixed it. (I noticed many violations of the official Javadoc style, but this is not the right pull request to fix them.) ------------- PR: https://git.openjdk.org/jdk/pull/10029 From duke at openjdk.org Sun Sep 25 17:04:39 2022 From: duke at openjdk.org (SWinxy) Date: Sun, 25 Sep 2022 17:04:39 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: Message-ID: On Thu, 25 Aug 2022 15:35:53 GMT, Michael Ernst wrote: > 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni Oh no what happened ![image](https://user-images.githubusercontent.com/8303399/192155378-4bb439fa-d27a-417d-8a3c-cf7f7329458f.png) src/java.xml/share/classes/com/sun/org/apache/xpath/internal/NodeSetDTM.java line 1053: > 1051: * fetch will take place at index 1. > 1052: * > 1053: * @return The current position index. Note that there is an inconsistent change here, albeit insignificant. In `NodeSet`, the capital 'The' is removed after the `@return`, while the lowercase 'the' is removed here. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From alanb at openjdk.org Sun Sep 25 17:04:40 2022 From: alanb at openjdk.org (Alan Bateman) Date: Sun, 25 Sep 2022 17:04:40 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: Message-ID: On Thu, 25 Aug 2022 15:35:53 GMT, Michael Ernst wrote: > 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni src/java.base/share/native/libzip/zlib/zlib.h line 756: > 754: If this is done, the old level and strategy will be applied to the data > 755: compressed before deflateParams(), and the new level and strategy will be > 756: applied to the data compressed after deflateParams(). This is an issue for the upstream zlib project, we probably don't want to change it here. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From lancea at openjdk.org Sun Sep 25 17:04:40 2022 From: lancea at openjdk.org (Lance Andersen) Date: Sun, 25 Sep 2022 17:04:40 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: Message-ID: On Tue, 30 Aug 2022 09:31:07 GMT, Alan Bateman wrote: >> 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni > > src/java.base/share/native/libzip/zlib/zlib.h line 756: > >> 754: If this is done, the old level and strategy will be applied to the data >> 755: compressed before deflateParams(), and the new level and strategy will be >> 756: applied to the data compressed after deflateParams(). > > This is an issue for the upstream zlib project, we probably don't want to change it here. Agree. See https://github.com/madler/zlib/commit/5752b171fd4cc96b8d1f9526ec1940199c6e9740 which is in the zlib development branch and addresses a wide range of typos. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From mernst at openjdk.org Sun Sep 25 17:04:40 2022 From: mernst at openjdk.org (Michael Ernst) Date: Sun, 25 Sep 2022 17:04:40 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: Message-ID: On Tue, 30 Aug 2022 09:55:18 GMT, Lance Andersen wrote: >> src/java.base/share/native/libzip/zlib/zlib.h line 756: >> >>> 754: If this is done, the old level and strategy will be applied to the data >>> 755: compressed before deflateParams(), and the new level and strategy will be >>> 756: applied to the data compressed after deflateParams(). >> >> This is an issue for the upstream zlib project, we probably don't want to change it here. > > Agree. See https://github.com/madler/zlib/commit/5752b171fd4cc96b8d1f9526ec1940199c6e9740 which is in the zlib development branch and addresses a wide range of typos. OK. I backed out the change to zlib. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From duke at openjdk.org Sun Sep 25 17:23:28 2022 From: duke at openjdk.org (SWinxy) Date: Sun, 25 Sep 2022 17:23:28 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: Message-ID: On Thu, 25 Aug 2022 15:35:53 GMT, Michael Ernst wrote: > 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni I'm not a committer (yet) so I can't [sponsor](https://openjdk.org/sponsor/) this PR, even though I would. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From jpai at openjdk.org Mon Sep 26 05:17:22 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Mon, 26 Sep 2022 05:17:22 GMT Subject: RFR: 8294193: Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory [v3] In-Reply-To: References: Message-ID: On Sat, 24 Sep 2022 06:11:21 GMT, Jaikiran Pai wrote: >> Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8294193? >> >> As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. >> tier1, tier2 and tier3 tests passed with this change. > > Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: > > review comments for test Thank you for the review, Alan. ------------- PR: https://git.openjdk.org/jdk/pull/10383 From jpai at openjdk.org Mon Sep 26 05:20:18 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Mon, 26 Sep 2022 05:20:18 GMT Subject: Integrated: 8294193: Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory In-Reply-To: References: Message-ID: On Thu, 22 Sep 2022 06:34:53 GMT, Jaikiran Pai wrote: > Can I please get a review of this change which proposes to fix the issue noted in https://bugs.openjdk.org/browse/JDK-8294193? > > As discussed in a related PR https://github.com/openjdk/jdk/pull/10266#issuecomment-1246800456, this is a genuine bug in the `java.nio.file.Files.createDirectories` implementation. The commit in this PR fixes that by making sure that links are followed when checking if a `Path` is a directory during the creation of directories in the `createDirectories` implementation. The `private` method which is changed in this PR is only used in the `Files.createDirectories` and no other place, so this change should not impact any other code. Additionally a new test has been added to reproduce the issue and verify the fix. > tier1, tier2 and tier3 tests passed with this change. This pull request has now been integrated. Changeset: 169a5d48 Author: Jaikiran Pai URL: https://git.openjdk.org/jdk/commit/169a5d48afbc6627f36a768c17c2a5e56219d9c7 Stats: 161 lines in 3 files changed: 124 ins; 34 del; 3 mod 8294193: Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory Reviewed-by: alanb ------------- PR: https://git.openjdk.org/jdk/pull/10383 From david.holmes at oracle.com Mon Sep 26 07:35:46 2022 From: david.holmes at oracle.com (David Holmes) Date: Mon, 26 Sep 2022 17:35:46 +1000 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? Message-ID: <42897c3c-d868-66a0-28fe-8fb156c74563@oracle.com> Man Cao manc at google.com on Fri Sep 23 22:24:21 UTC 2022 wrote: > Thank you both for the feedback. > > > I realise the focus of JDK-8294052 has narrowed > > since it was created but I think it would be fair to say that it would > > probably require going through them case by case to see if it make sense. > > Yes. I completely agree that the OOMEs should be analyzed case by case. > Also completely agree that the OOMEs for large array allocations do not > need to handle these JVM flags. > > > I think there is a case for triggering a heap dump or doing the other > > OOME actions when a direct buffer can't be allocated. > > ... > > It sounds like I could proceed sending the RFR to support these JVM flags > for direct memory OOME? > In our case, the application sets -XX:MaxDirectMemorySize and it ran out of > this limit. > While bumping up MaxDirectMemorySize could solve the problem, we'd like to > understand where the increase comes from. We already monitor direct memory > usage via BufferPoolMXBean, but it is not sufficient to dig into the root > cause. We really heap dumps to understand the root cause. If all you really need is heap dumps then aren't there other ways to get those via the management API (that you are already using)? I really do not want to turn these VM flags into an exported API for use by JDK code if it can be avoided (they are crude and basic and minimally tested). If that kind of functionality is needed then I'd prefer to see a JDK-side API to provide it in a cleaner, more supported way. David ----- > -Man From alanb at openjdk.org Mon Sep 26 16:13:02 2022 From: alanb at openjdk.org (Alan Bateman) Date: Mon, 26 Sep 2022 16:13:02 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow Message-ID: BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket address to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. ------------- Commit messages: - Initial commit Changes: https://git.openjdk.org/jdk/pull/10427/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10427&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8294375 Stats: 602 lines in 2 files changed: 125 ins; 311 del; 166 mod Patch: https://git.openjdk.org/jdk/pull/10427.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10427/head:pull/10427 PR: https://git.openjdk.org/jdk/pull/10427 From joehw at openjdk.org Mon Sep 26 16:16:34 2022 From: joehw at openjdk.org (Joe Wang) Date: Mon, 26 Sep 2022 16:16:34 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni In-Reply-To: References: Message-ID: On Thu, 25 Aug 2022 15:35:53 GMT, Michael Ernst wrote: > 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni We have discussed several times in the past that we don't touch upstream (in this case, Apache) code unless the change is essential. In general, if you fix a bug in a class, it's fine to fix other non-essential things within the class, such as code styles, javadocs, typos as in this case, etc. Otherwise, I would suggest you contribute such changes to the upstream projects. Thanks. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From mernst at openjdk.org Mon Sep 26 16:51:36 2022 From: mernst at openjdk.org (Michael Ernst) Date: Mon, 26 Sep 2022 16:51:36 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni [v2] In-Reply-To: References: Message-ID: > 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni Michael Ernst has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits: - Reinstate typos in Apache code that is copied into the JDK - Merge ../jdk-openjdk into typos-typos - Remove file that was removed upstream - Fix inconsistency in capitalization - Undo change in zlip - Fix typos ------------- Changes: https://git.openjdk.org/jdk/pull/10029/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10029&range=01 Stats: 85 lines in 76 files changed: 0 ins; 0 del; 85 mod Patch: https://git.openjdk.org/jdk/pull/10029.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10029/head:pull/10029 PR: https://git.openjdk.org/jdk/pull/10029 From joehw at openjdk.org Mon Sep 26 17:04:25 2022 From: joehw at openjdk.org (Joe Wang) Date: Mon, 26 Sep 2022 17:04:25 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni [v2] In-Reply-To: References: Message-ID: On Mon, 26 Sep 2022 16:51:36 GMT, Michael Ernst wrote: >> 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni > > Michael Ernst has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits: > > - Reinstate typos in Apache code that is copied into the JDK > - Merge ../jdk-openjdk into typos-typos > - Remove file that was removed upstream > - Fix inconsistency in capitalization > - Undo change in zlip > - Fix typos Thanks for the update! ------------- PR: https://git.openjdk.org/jdk/pull/10029 From Alan.Bateman at oracle.com Mon Sep 26 17:31:22 2022 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 26 Sep 2022 18:31:22 +0100 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? In-Reply-To: <42897c3c-d868-66a0-28fe-8fb156c74563@oracle.com> References: <42897c3c-d868-66a0-28fe-8fb156c74563@oracle.com> Message-ID: <02f14601-900a-ffa1-45c1-6500e9db84e7@oracle.com> On 26/09/2022 08:35, David Holmes wrote: > : > > If all you really need is heap dumps then aren't there other ways to > get those via the management API (that you are already using)? > > I really do not want to turn these VM flags into an exported API for > use by JDK code if it can be avoided (they are crude and basic and > minimally tested). If that kind of functionality is needed then I'd > prefer to see a JDK-side API to provide it in a cleaner, more > supported way. We added the built-in heap dumper and HeapDumpOnOutOfMemoryError in JDK 6 to help with post mortem diagnosability of OOME. They have been supported and documented (e.g. troubleshooting guide) and there are several tools that support reading the .hprof format. It may be rudimentary but the option to enable it seems to be widely used. Yes, you can use the management APIs and BufferPoolMXBean to monitor the use of direct memory and use the HotSpotDiagnosticMXBean "dumpHeap" operation to generate a heap dump at any time. You can use another HotSpotDiagnosticMXBean operation to enable the HeapDumpOnOutOfMemoryError at runtime but it's not going to trigger if the OOME is triggered trying to allocate direct memory. I don't know if you have a JDK-specific API in mind but I think the discussion here is about having a way to trigger a heap dump (and maybe other OOME actions) when allocating direct memory fails. -Alan From lancea at openjdk.org Mon Sep 26 19:38:20 2022 From: lancea at openjdk.org (Lance Andersen) Date: Mon, 26 Sep 2022 19:38:20 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni [v2] In-Reply-To: References: Message-ID: <3PJ-Gv4AW5nTD3gcQ1mwZH5C0jFo09zD6879B3jlkY4=.05beeb4b-498d-40fb-8514-244ac0ef0772@github.com> On Mon, 26 Sep 2022 16:51:36 GMT, Michael Ernst wrote: >> 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni > > Michael Ernst has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits: > > - Reinstate typos in Apache code that is copied into the JDK > - Merge ../jdk-openjdk into typos-typos > - Remove file that was removed upstream > - Fix inconsistency in capitalization > - Undo change in zlip > - Fix typos Thank you for your efforts on the cleaning up these typos I would suggest breaking these changes into separate PRs by team as there seems to be changes to hotspot, core-libs, and client to make it easier to review. I realize the changes are fairly trivial but this will most likely expedite getting your changes back ------------- PR: https://git.openjdk.org/jdk/pull/10029 From aturbanov at openjdk.org Mon Sep 26 21:08:30 2022 From: aturbanov at openjdk.org (Andrey Turbanov) Date: Mon, 26 Sep 2022 21:08:30 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v5] In-Reply-To: References: Message-ID: On Thu, 22 Sep 2022 08:57:21 GMT, Maxim Kartashev wrote: >> This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. >> >> Some things of note: >> * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. >> * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). >> * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). >> * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). >> * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. >> >> This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. >> >> ### Testing >> >> * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). >> * Also verified that new tests pass on Linux and Windows. >> * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. > > Maxim Kartashev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains ten commits: > > - Merge master > - Minor changes based on review comments > - Removed the change unnecessary without recursive watch > - Removed support for recursive watch > - Reformatted native code and removed jlong and pointer casts > - Removed the dependency on provider(Path) > - More robust way of obtaining run loop reference > - CFRunLoopThread to use InnocuousThread > - 8293067: (fs) Implement WatchService using system library (macOS) src/java.base/macosx/classes/sun/nio/fs/MacOSXWatchService.java line 313: > 311: private static final long kFSEventStreamEventFlagRootChanged = 0x00000020; > 312: > 313: private final static Path relativeRootPath = theFS.getPath(""); let's use blessed modifiers order: `private static final` ------------- PR: https://git.openjdk.org/jdk/pull/10140 From bpb at openjdk.org Mon Sep 26 21:57:12 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Mon, 26 Sep 2022 21:57:12 GMT Subject: RFR: 8294399: (ch) Refactor some methods out of sun.nio.ch.UnixFileDispatcherImpl Message-ID: Interpose a new class `sun.nio.ch.UnixDispatcher` between `sun.nio.ch.NativeDispatcher` and its current subclasses `sun.nio.ch.DatagramDispatcher` and `sun.nio.ch.SocketDispatcher`; refactor some methods out of `sun.nio.ch.UnixFileDispatcherImpl` into this updated hierarchy. ------------- Commit messages: - 8294399: (ch) Refactor some methods out of sun.nio.ch.UnixFileDispatcherImpl Changes: https://git.openjdk.org/jdk/pull/10434/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10434&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8294399 Stats: 231 lines in 9 files changed: 158 ins; 60 del; 13 mod Patch: https://git.openjdk.org/jdk/pull/10434.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10434/head:pull/10434 PR: https://git.openjdk.org/jdk/pull/10434 From bpb at openjdk.org Mon Sep 26 22:17:23 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Mon, 26 Sep 2022 22:17:23 GMT Subject: RFR: 8294399: (ch) Refactor some methods out of sun.nio.ch.UnixFileDispatcherImpl [v2] In-Reply-To: References: Message-ID: > Interpose a new class `sun.nio.ch.UnixDispatcher` between `sun.nio.ch.NativeDispatcher` and its current subclasses `sun.nio.ch.DatagramDispatcher` and `sun.nio.ch.SocketDispatcher`; refactor some methods out of `sun.nio.ch.UnixFileDispatcherImpl` into this updated hierarchy. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8294399: Correct copyright years ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10434/files - new: https://git.openjdk.org/jdk/pull/10434/files/92dfca5f..a16bc10a Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10434&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10434&range=00-01 Stats: 3 lines in 3 files changed: 0 ins; 0 del; 3 mod Patch: https://git.openjdk.org/jdk/pull/10434.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10434/head:pull/10434 PR: https://git.openjdk.org/jdk/pull/10434 From manc at google.com Tue Sep 27 04:16:01 2022 From: manc at google.com (Man Cao) Date: Mon, 26 Sep 2022 21:16:01 -0700 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? In-Reply-To: <30ea76f7-cdfe-b357-8427-0fd0627dd3fa@oracle.com> References: <7e024658-20bc-fe9a-38ef-f668ea5a4e5e@oracle.com> <30ea76f7-cdfe-b357-8427-0fd0627dd3fa@oracle.com> Message-ID: > > I don't have any objection to this. I only mentioned BufferPoolMXBean to > show that memory for direct and mapped buffers is already exposed for > monitoring purposes. Also direct buffers are a clear case where the > memory is kept alive by live objects and at the same time have a runtime > imposed limit. With the foreign memory API (Project Panama) it means > there will also be memory segments that are part of lifecycles managed > by memory sessions - in time, these will also be resources that may need > to be monitored and maybe trigger heap dumps / OOME actions too. Great! Thanks for the feedback. I'll proceed with the RFR. +1 for the foreign memory API's case as well. In JDK-8294052's comment, I too mentioned OOME from jdk/internal/foreign/ArenaAllocator.java could benefit from supporting these flags. -Man On Sun, Sep 25, 2022 at 1:10 AM Alan Bateman wrote: > On 23/09/2022 23:24, Man Cao wrote: > > : > > > > Yes. I completely agree that the OOMEs should be analyzed case by case. > > Also completely agree that the OOMEs for large array allocations do > > not need to handle these JVM flags. > > I think OOME has been over used and if we could go back in time then > maybe things would be different and OOME would not be thrown for some of > these cases. > > > > : > > > > It sounds like I could proceed sending the RFR to support these JVM > > flags for direct memory OOME? > > In our case, the application sets -XX:MaxDirectMemorySize and it ran > > out of this limit. > > While bumping up MaxDirectMemorySize could solve the problem, we'd > > like to understand where the increase comes from. We already monitor > > direct memory usage via BufferPoolMXBean, but it is not sufficient to > > dig into the root cause. We really heap dumps to understand the root > > cause. > > > I don't have any objection to this. I only mentioned BufferPoolMXBean to > show that memory for direct and mapped buffers is already exposed for > monitoring purposes. Also direct buffers are a clear case where the > memory is kept alive by live objects and at the same time have a runtime > imposed limit. With the foreign memory API (Project Panama) it means > there will also be memory segments that are part of lifecycles managed > by memory sessions - in time, these will also be resources that may need > to be monitored and maybe trigger heap dumps / OOME actions too. > > -Alan > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From manc at google.com Tue Sep 27 06:29:25 2022 From: manc at google.com (Man Cao) Date: Mon, 26 Sep 2022 23:29:25 -0700 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? In-Reply-To: <02f14601-900a-ffa1-45c1-6500e9db84e7@oracle.com> References: <42897c3c-d868-66a0-28fe-8fb156c74563@oracle.com> <02f14601-900a-ffa1-45c1-6500e9db84e7@oracle.com> Message-ID: > If all you really need is heap dumps then aren't there other ways to get > those via the management API (that you are already using)? As Alan pointed out, for the direct memory OOME case, the timing of triggering the heap dump is difficult without HeapDumpOnOutOfMemoryError. If the direct memory API offers a callback that executes a user-provided function on each allocation or before throwing OOME, that would be easier for user code to trigger a heap dump before OOME. However, supporting HeapDumpOnOutOfMemoryError and related flags seems much easier than modifying the direct memory API. > I really do not want to turn these VM flags into an exported API for use > by JDK code if it can be avoided (they are crude and basic and minimally > tested). If that kind of functionality is needed then I'd prefer to see > a JDK-side API to provide it in a cleaner, more supported way. Even if we provide a JDK-side API (like the callback idea mentioned above), I suppose the API would not respect HeapDumpOnOutOfMemoryError and related JVM flags? As Alan (and [~stuefe] and [~xliu] from JDK-8257790) mentioned, those JVM flags are well-known and users would expect them to work when they need to debug these real resource-exhaustion OOMEs. For the "crude and basic and minimally tested" argument, I don't think it is a reason to not support them for these useful cases. These OOME flags are extremely widely used in our production jobs (and possibly also for SAP and Amazon). I'm confident the community will help maintain these flags and make sure they function as expected. -Man -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.holmes at oracle.com Tue Sep 27 07:21:46 2022 From: david.holmes at oracle.com (David Holmes) Date: Tue, 27 Sep 2022 17:21:46 +1000 Subject: Should OutOfMemoryError from NIO direct memory honor JVM flags for handling OOME? In-Reply-To: <02f14601-900a-ffa1-45c1-6500e9db84e7@oracle.com> References: <42897c3c-d868-66a0-28fe-8fb156c74563@oracle.com> <02f14601-900a-ffa1-45c1-6500e9db84e7@oracle.com> Message-ID: On 27/09/2022 3:31 am, Alan Bateman wrote: > On 26/09/2022 08:35, David Holmes wrote: >> : >> >> If all you really need is heap dumps then aren't there other ways to >> get those via the management API (that you are already using)? >> >> I really do not want to turn these VM flags into an exported API for >> use by JDK code if it can be avoided (they are crude and basic and >> minimally tested). If that kind of functionality is needed then I'd >> prefer to see a JDK-side API to provide it in a cleaner, more >> supported way. > > We added the built-in heap dumper and HeapDumpOnOutOfMemoryError in JDK > 6 to help with post mortem diagnosability of OOME. They have been > supported and documented (e.g. troubleshooting guide) and there are > several tools that support reading the .hprof format. It may be > rudimentary but the option to enable it seems to be widely used. This is the way I see things (and I'm taking a bit of "poetic license" here). The Java language provided the OOME to report Java OOM conditions, but didn't provide the diagnostic tools to help with understanding the origins/causes. So we provided a mechanism in the VM to help with this. And yes there are also some Java-level tools to give some aid here. But people also want to diagnose other resource allocation failures, and again there are no diagnostic tools to assist them. So the natural first reaction is "hey lets just make this VM OOM stuff work for this situation as well". My reaction is "hey, maybe the JDK should be providing a more structured/consistent/supported mechanism for getting diagnostics for arbitrary resource allocation failures?". Obviously the VM has to hook into that - and arguably what the VM should be doing is just calling back up to Java to allow it to execute a registered callback that can do whatever is needed based on some diagnostic properties. I'd much rather see a well-designed resource allocation failure diagnostic framework added to the JDK, than crudely provide a JVM_OnAllocationFailure method that then examines the various VM flags. Cheers, David ----- > Yes, you can use the management APIs and BufferPoolMXBean to monitor the > use of direct memory and use the HotSpotDiagnosticMXBean "dumpHeap" > operation to generate a heap dump at any time. You can use another > HotSpotDiagnosticMXBean operation to enable the > HeapDumpOnOutOfMemoryError at runtime but it's not going to trigger if > the OOME is triggered trying to allocate direct memory. > > I don't know if you have a JDK-specific API in mind but I think the > discussion here is about having a way to trigger a heap dump (and maybe > other OOME actions) when allocating direct memory fails. > > -Alan From mkartashev at openjdk.org Tue Sep 27 07:25:58 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Tue, 27 Sep 2022 07:25:58 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v6] In-Reply-To: References: Message-ID: > This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. > > Some things of note: > * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. > * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). > * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). > * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). > * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. > > This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. > > ### Testing > > * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). > * Also verified that new tests pass on Linux and Windows. > * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. Maxim Kartashev has updated the pull request incrementally with one additional commit since the last revision: Straightened the order of field modifiers ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10140/files - new: https://git.openjdk.org/jdk/pull/10140/files/c9ff697e..d834262f Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=05 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=04-05 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jdk/pull/10140.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10140/head:pull/10140 PR: https://git.openjdk.org/jdk/pull/10140 From mkartashev at openjdk.org Tue Sep 27 07:26:04 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Tue, 27 Sep 2022 07:26:04 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v5] In-Reply-To: References: Message-ID: On Mon, 26 Sep 2022 21:04:31 GMT, Andrey Turbanov wrote: >> Maxim Kartashev has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains ten commits: >> >> - Merge master >> - Minor changes based on review comments >> - Removed the change unnecessary without recursive watch >> - Removed support for recursive watch >> - Reformatted native code and removed jlong and pointer casts >> - Removed the dependency on provider(Path) >> - More robust way of obtaining run loop reference >> - CFRunLoopThread to use InnocuousThread >> - 8293067: (fs) Implement WatchService using system library (macOS) > > src/java.base/macosx/classes/sun/nio/fs/MacOSXWatchService.java line 313: > >> 311: private static final long kFSEventStreamEventFlagRootChanged = 0x00000020; >> 312: >> 313: private final static Path relativeRootPath = theFS.getPath(""); > > let's use blessed modifiers order: `private static final` Sure! Fixed this one. ------------- PR: https://git.openjdk.org/jdk/pull/10140 From jpai at openjdk.org Tue Sep 27 11:52:23 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Tue, 27 Sep 2022 11:52:23 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow In-Reply-To: References: Message-ID: On Mon, 26 Sep 2022 15:35:28 GMT, Alan Bateman wrote: > BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. > > BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. > > In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. test/jdk/java/nio/channels/vthread/BlockingChannelOps.java line 206: > 204: assertTrue(n > 0); > 205: bb.clear(); > 206: } Hello Alan, since `AsynchronousCloseException` is expected here, should we add a `fail` after this loop if we don't receive that exception ------------- PR: https://git.openjdk.org/jdk/pull/10427 From prappo at openjdk.org Tue Sep 27 11:52:57 2022 From: prappo at openjdk.org (Pavel Rappo) Date: Tue, 27 Sep 2022 11:52:57 GMT Subject: RFR: 8294377: Prepare to stop auto-inheriting documentation for subclasses of exceptions whose documentation is inherited Message-ID: This adds exception documentation to JDK methods that would otherwise lose that documentation once JDK-8287796 is integrated. While adding this exception documentation now does not change [^1] the JDK API Documentation, it will automatically counterbalance the effect that JDK-8287796 will have. JDK-8287796 seeks to remove an old, undocumented, and esoteric javadoc feature that cannot be suppressed [^2]. The feature is so little known about that IDEs either do not implement it correctly or do not implement it at all, thus rendering documentation differently from how the javadoc tool renders it. That feature was introduced in JDK-4947455 and manifests as follows. If a method inherits documentation for an exception, along with that documentation the method automatically inherits documentation for all exceptions that are subclasses of that former exception and are documented in an overridden method. To illustrate that behavior, consider the following example. A method `Subtype.m` inherits documentation for `SuperException`, while the overridden method `Supertype.m` documents `SuperException`, `SubExceptionA` and `SubExceptionB`, where the latter two exceptions are subclasses of the former exception: public class Subtype extends Supertype { @Override public void m() throws SuperException { ... public class Supertype { /** * @throws SuperException general problem * @throws SubExceptionA specific problem A * @throws SubExceptionB specific problem B */ public void m() throws SuperException { ... public class SuperException extends Exception { ... public class SubExceptionA extends SuperException { ... public class SubExceptionB extends SuperException { ... For the above example, the API Documentation for `Subtype.m` will contain documentation for all three exceptions; the page fragment for `Subtype.m` in "Method Details" will look like this: public void m() throws SuperException Overrides: m in class Supertype Throws: SuperException - general problem SubExceptionA - specific problem A SubExceptionB - specific problem B It works for checked and unchecked exceptions, for methods in classes and interfaces. If it's the first time you hear about that behavior and this change affects your area, it's a good opportunity to reflect on whether the exception documentation this change adds is really needed or you were simply unaware of the fact that it was automatically added before. If exception documentation is not needed, please comment on this PR. Otherwise, consider approving it. Keep in mind that if some exception documentation is not needed, **leaving it out** from this PR might require a CSR. [^1]: Aside from insignificant changes on class-use and index pages. There's one relatively significant change. This change is in jdk.jshell, where some methods disappeared from "Methods declared in ..." section and other irregularities. We are aware of this and have filed JDK-8291803 to fix the root cause. [^2]: In reality, the feature can be suppressed, but it looks like a bug rather than intentional design. If we legitimize the feature and its suppression behavior, the model for exception documentation inheritance might become much more complicated. ------------- Commit messages: - Initial commit Changes: https://git.openjdk.org/jdk/pull/10449/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10449&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8294377 Stats: 322 lines in 15 files changed: 311 ins; 0 del; 11 mod Patch: https://git.openjdk.org/jdk/pull/10449.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10449/head:pull/10449 PR: https://git.openjdk.org/jdk/pull/10449 From jpai at openjdk.org Tue Sep 27 11:58:32 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Tue, 27 Sep 2022 11:58:32 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow In-Reply-To: References: Message-ID: On Mon, 26 Sep 2022 15:35:28 GMT, Alan Bateman wrote: > BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. > > BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. > > In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. test/jdk/java/nio/channels/vthread/BlockingChannelOps.java line 236: > 234: assertTrue(n > 0); > 235: bb.clear(); > 236: } Same here? Should we add a `fail` if we don't receive a `ClosedByInterruptException`? ------------- PR: https://git.openjdk.org/jdk/pull/10427 From alanb at openjdk.org Tue Sep 27 12:04:24 2022 From: alanb at openjdk.org (Alan Bateman) Date: Tue, 27 Sep 2022 12:04:24 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow In-Reply-To: References: Message-ID: On Tue, 27 Sep 2022 11:50:16 GMT, Jaikiran Pai wrote: >> BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. >> >> BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. >> >> In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. > > test/jdk/java/nio/channels/vthread/BlockingChannelOps.java line 206: > >> 204: assertTrue(n > 0); >> 205: bb.clear(); >> 206: } > > Hello Alan, since `AsynchronousCloseException` is expected here, should we add a `fail` after this loop if we don't receive that exception It's an infinite loop as write may need to be called many times before it blocks. So you can't put a `fail` after the loop as it will never be executed. If another exception is thrown then the test will fail. ------------- PR: https://git.openjdk.org/jdk/pull/10427 From prappo at openjdk.org Tue Sep 27 12:14:23 2022 From: prappo at openjdk.org (Pavel Rappo) Date: Tue, 27 Sep 2022 12:14:23 GMT Subject: RFR: 8294377: Prepare to stop auto-inheriting documentation for subclasses of exceptions whose documentation is inherited [v2] In-Reply-To: References: Message-ID: > This adds exception documentation to JDK methods that would otherwise lose that documentation once JDK-8287796 is integrated. While adding this exception documentation now does not change [^1] the JDK API Documentation, it will automatically counterbalance the effect that JDK-8287796 will have. > > JDK-8287796 seeks to remove an old, undocumented, and esoteric javadoc feature that cannot be suppressed [^2]. The feature is so little known about that IDEs either do not implement it correctly or do not implement it at all, thus rendering documentation differently from how the javadoc tool renders it. > > That feature was introduced in JDK-4947455 and manifests as follows. If a method inherits documentation for an exception, along with that documentation the method automatically inherits documentation for all exceptions that are subclasses of that former exception and are documented in an overridden method. > > To illustrate that behavior, consider the following example. A method `Subtype.m` inherits documentation for `SuperException`, while the overridden method `Supertype.m` documents `SuperException`, `SubExceptionA` and `SubExceptionB`, where the latter two exceptions are subclasses of the former exception: > > public class Subtype extends Supertype { > > @Override > public void m() throws SuperException { > ... > > public class Supertype { > > /** > * @throws SuperException general problem > * @throws SubExceptionA specific problem A > * @throws SubExceptionB specific problem B > */ > public void m() throws SuperException { > ... > > public class SuperException extends Exception { > ... > > public class SubExceptionA extends SuperException { > ... > > public class SubExceptionB extends SuperException { > ... > > For the above example, the API Documentation for `Subtype.m` will contain documentation for all three exceptions; the page fragment for `Subtype.m` in "Method Details" will look like this: > > public void m() > throws SuperException > > Overrides: > m in class Supertype > Throws: > SuperException - general problem > SubExceptionA - specific problem A > SubExceptionB - specific problem B > > It works for checked and unchecked exceptions, for methods in classes and interfaces. > > > If it's the first time you hear about that behavior and this change affects your area, it's a good opportunity to reflect on whether the exception documentation this change adds is really needed or you were simply unaware of the fact that it was automatically added before. If exception documentation is not needed, please comment on this PR. Otherwise, consider approving it. > > Keep in mind that if some exception documentation is not needed, **leaving it out** from this PR might require a CSR. > > > [^1]: Aside from insignificant changes on class-use and index pages. There's one relatively significant change. This change is in jdk.jshell, where some methods disappeared from "Methods declared in ..." section and other irregularities. We are aware of this and have filed JDK-8291803 to fix the root cause. > [^2]: In reality, the feature can be suppressed, but it looks like a bug rather than intentional design. If we legitimize the feature and its suppression behavior, the model for exception documentation inheritance might become much more complicated. Pavel Rappo has updated the pull request incrementally with one additional commit since the last revision: revert an extraneous change to jdk.javadoc ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10449/files - new: https://git.openjdk.org/jdk/pull/10449/files/87bdbff4..d303f5ec Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10449&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10449&range=00-01 Stats: 2 lines in 1 file changed: 0 ins; 1 del; 1 mod Patch: https://git.openjdk.org/jdk/pull/10449.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10449/head:pull/10449 PR: https://git.openjdk.org/jdk/pull/10449 From jpai at openjdk.org Tue Sep 27 12:19:51 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Tue, 27 Sep 2022 12:19:51 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow In-Reply-To: References: Message-ID: On Tue, 27 Sep 2022 12:02:06 GMT, Alan Bateman wrote: >> test/jdk/java/nio/channels/vthread/BlockingChannelOps.java line 206: >> >>> 204: assertTrue(n > 0); >>> 205: bb.clear(); >>> 206: } >> >> Hello Alan, since `AsynchronousCloseException` is expected here, should we add a `fail` after this loop if we don't receive that exception > > It's an infinite loop as write may need to be called many times before it blocks. So you can't put a `fail` after the loop as it will never be executed. If another exception is thrown then the test will fail. You are right; I overlooked the part that this `for` loop is infinite. So, for this test method to complete, the only way out of that `for` loop is some exception getting thrown and we then catch the expected exception and ignore it or let the unexpected exception propagate into a test failure. So yes, the current code looks fine. ------------- PR: https://git.openjdk.org/jdk/pull/10427 From jpai at openjdk.org Tue Sep 27 12:19:52 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Tue, 27 Sep 2022 12:19:52 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow In-Reply-To: References: Message-ID: On Mon, 26 Sep 2022 15:35:28 GMT, Alan Bateman wrote: > BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. > > BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. > > In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. test/jdk/java/nio/channels/vthread/BlockingChannelOps.java line 643: > 641: Pipe.SourceChannel source = p.source()) { > 642: > 643: // delaye write from sink Minor typo - `delaye` instead of `delayed` ------------- PR: https://git.openjdk.org/jdk/pull/10427 From dfuchs at openjdk.org Tue Sep 27 13:47:31 2022 From: dfuchs at openjdk.org (Daniel Fuchs) Date: Tue, 27 Sep 2022 13:47:31 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow In-Reply-To: References: Message-ID: On Mon, 26 Sep 2022 15:35:28 GMT, Alan Bateman wrote: > BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. > > BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. > > In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. test/jdk/java/net/vthread/BlockingSocketOps.java line 714: > 712: } > 713: > 714: static void readToEOF(InputStream in) throws IOException { just curious: isn't that just `in.readAllBytes()`? Oh - I see. You don't want to accumulate all the bytes. An alternative would be `in.transferTo(OutputStream.nullOutputStream())` ------------- PR: https://git.openjdk.org/jdk/pull/10427 From rriggs at openjdk.org Tue Sep 27 13:59:28 2022 From: rriggs at openjdk.org (Roger Riggs) Date: Tue, 27 Sep 2022 13:59:28 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow In-Reply-To: References: Message-ID: On Tue, 27 Sep 2022 13:35:41 GMT, Daniel Fuchs wrote: >> BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. >> >> BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. >> >> In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. > > test/jdk/java/net/vthread/BlockingSocketOps.java line 714: > >> 712: } >> 713: >> 714: static void readToEOF(InputStream in) throws IOException { > > just curious: isn't that just `in.readAllBytes()`? > Oh - I see. You don't want to accumulate all the bytes. > An alternative would be `in.transferTo(OutputStream.nullOutputStream())` Or perhaps `InputStream.skip(MAX_INT)`. ------------- PR: https://git.openjdk.org/jdk/pull/10427 From alanb at openjdk.org Tue Sep 27 13:59:29 2022 From: alanb at openjdk.org (Alan Bateman) Date: Tue, 27 Sep 2022 13:59:29 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow In-Reply-To: References: Message-ID: On Tue, 27 Sep 2022 13:55:14 GMT, Roger Riggs wrote: >> test/jdk/java/net/vthread/BlockingSocketOps.java line 714: >> >>> 712: } >>> 713: >>> 714: static void readToEOF(InputStream in) throws IOException { >> >> just curious: isn't that just `in.readAllBytes()`? >> Oh - I see. You don't want to accumulate all the bytes. >> An alternative would be `in.transferTo(OutputStream.nullOutputStream())` > > Or perhaps `InputStream.skip(MAX_INT)`. Yes, this would work too. We just need something to read the input stream to EOF so that the connection peer reads -1/EOF. ------------- PR: https://git.openjdk.org/jdk/pull/10427 From alanb at openjdk.org Tue Sep 27 15:35:34 2022 From: alanb at openjdk.org (Alan Bateman) Date: Tue, 27 Sep 2022 15:35:34 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow [v2] In-Reply-To: References: Message-ID: > BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. > > BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. > > In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. Alan Bateman has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision: - Change testSocketWrite1 to use in.transfer, and fix typo in comment - Merge - Initial commit ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10427/files - new: https://git.openjdk.org/jdk/pull/10427/files/a1a23d1d..54da8dee Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10427&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10427&range=00-01 Stats: 2990 lines in 97 files changed: 2464 ins; 347 del; 179 mod Patch: https://git.openjdk.org/jdk/pull/10427.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10427/head:pull/10427 PR: https://git.openjdk.org/jdk/pull/10427 From mernst at openjdk.org Tue Sep 27 23:16:16 2022 From: mernst at openjdk.org (Michael Ernst) Date: Tue, 27 Sep 2022 23:16:16 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni [v2] In-Reply-To: <3PJ-Gv4AW5nTD3gcQ1mwZH5C0jFo09zD6879B3jlkY4=.05beeb4b-498d-40fb-8514-244ac0ef0772@github.com> References: <3PJ-Gv4AW5nTD3gcQ1mwZH5C0jFo09zD6879B3jlkY4=.05beeb4b-498d-40fb-8514-244ac0ef0772@github.com> Message-ID: <1UZcv-X4dxQHlgj8b5B2sGJ9NpxWrOp1EwbWeazjj2I=.1cf2b14f-7064-47b6-8d25-887732020bb0@github.com> On Mon, 26 Sep 2022 19:36:02 GMT, Lance Andersen wrote: > Thank you for your efforts on the cleaning up these typos > > I would suggest breaking these changes into separate PRs by team as there seems to be changes to hotspot, core-libs, and client to make it easier to review. > > I realize the changes are fairly trivial but this will most likely expedite getting your changes back Feel free to break up the pull request if that is what is needed to free it from red tape. Thanks. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From jpai at openjdk.org Wed Sep 28 09:04:33 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Wed, 28 Sep 2022 09:04:33 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow [v2] In-Reply-To: References: Message-ID: On Tue, 27 Sep 2022 15:35:34 GMT, Alan Bateman wrote: >> BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. >> >> BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. >> >> In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. > > Alan Bateman has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision: > > - Change testSocketWrite1 to use in.transfer, and fix typo in comment > - Merge > - Initial commit test/jdk/java/net/vthread/BlockingSocketOps.java line 722: > 720: /** > 721: * Runs the given task asynchronously after the current virtual thread has > 722: * parked, or after a max wait time, whichever is first. Does this comment need an update to match the current implementation? From what I can see in the current implementation of `runAfterParkedAsync`, in this PR, it doesn't have a "max wait time" construct before triggering the task. ------------- PR: https://git.openjdk.org/jdk/pull/10427 From jpai at openjdk.org Wed Sep 28 09:49:28 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Wed, 28 Sep 2022 09:49:28 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow [v2] In-Reply-To: References: Message-ID: On Tue, 27 Sep 2022 15:35:34 GMT, Alan Bateman wrote: >> BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. >> >> BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. >> >> In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. > > Alan Bateman has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision: > > - Change testSocketWrite1 to use in.transfer, and fix typo in comment > - Merge > - Initial commit > BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. That's a really good improvement on the testing time. I gave this PR a try on my local Mac setup and as you note, each `@run` of the test now completes between 2 to 4 seconds for me. Now that these tests complete fast, do you think we should now remove all the `/timeout=300` from each of the test definitions? ------------- PR: https://git.openjdk.org/jdk/pull/10427 From alanb at openjdk.org Wed Sep 28 10:01:18 2022 From: alanb at openjdk.org (Alan Bateman) Date: Wed, 28 Sep 2022 10:01:18 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow [v2] In-Reply-To: References: Message-ID: On Wed, 28 Sep 2022 09:45:39 GMT, Jaikiran Pai wrote: > Now that these tests complete fast, do you think we should now remove all the `/timeout=300` from each of the test definitions? I'll have to check. The reason for /timeout on some of these tests is because most of the testing has been with debug builds rather than release builds. > test/jdk/java/net/vthread/BlockingSocketOps.java line 722: > >> 720: /** >> 721: * Runs the given task asynchronously after the current virtual thread has >> 722: * parked, or after a max wait time, whichever is first. > > Does this comment need an update to match the current implementation? From what I can see in the current implementation of `runAfterParkedAsync`, in this PR, it doesn't have a "max wait time" construct before triggering the task. Well spotted, that comment is out of date now. ------------- PR: https://git.openjdk.org/jdk/pull/10427 From alanb at openjdk.org Wed Sep 28 11:08:26 2022 From: alanb at openjdk.org (Alan Bateman) Date: Wed, 28 Sep 2022 11:08:26 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow [v3] In-Reply-To: References: Message-ID: > BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. > > BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. > > In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. Alan Bateman has updated the pull request incrementally with one additional commit since the last revision: Drop /timeout from run tags. Fix out of date comment ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10427/files - new: https://git.openjdk.org/jdk/pull/10427/files/54da8dee..7097193e Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10427&range=02 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10427&range=01-02 Stats: 10 lines in 2 files changed: 0 ins; 2 del; 8 mod Patch: https://git.openjdk.org/jdk/pull/10427.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10427/head:pull/10427 PR: https://git.openjdk.org/jdk/pull/10427 From djelinski at openjdk.org Wed Sep 28 11:20:54 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Wed, 28 Sep 2022 11:20:54 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows Message-ID: Please review this test-only change that improves the execution speed of a few FileChannel tests: - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) ------------- Commit messages: - MapTest: avoid costly repetition - Simplify LargeMapTest - LargeMapTest: use sparse file - Test4g: use sparse files - Test6g: use sparse file - Test2g: write sparse file Changes: https://git.openjdk.org/jdk/pull/10464/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10464&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8294437 Stats: 153 lines in 5 files changed: 65 ins; 23 del; 65 mod Patch: https://git.openjdk.org/jdk/pull/10464.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10464/head:pull/10464 PR: https://git.openjdk.org/jdk/pull/10464 From jpai at openjdk.org Wed Sep 28 12:02:21 2022 From: jpai at openjdk.org (Jaikiran Pai) Date: Wed, 28 Sep 2022 12:02:21 GMT Subject: RFR: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow [v3] In-Reply-To: References: Message-ID: <14VbrWzM8jHhI1NWDt1fwSV3go1I8uG0vO08HVWHpkQ=.50049c3b-6110-46a3-94f2-f178afa9128f@github.com> On Wed, 28 Sep 2022 11:08:26 GMT, Alan Bateman wrote: >> BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. >> >> BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. >> >> In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. > > Alan Bateman has updated the pull request incrementally with one additional commit since the last revision: > > Drop /timeout from run tags. Fix out of date comment Thank you for those changes. The current state of this PR looks fine to me. ------------- Marked as reviewed by jpai (Reviewer). PR: https://git.openjdk.org/jdk/pull/10427 From alanb at openjdk.org Wed Sep 28 13:13:44 2022 From: alanb at openjdk.org (Alan Bateman) Date: Wed, 28 Sep 2022 13:13:44 GMT Subject: Integrated: 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow In-Reply-To: References: Message-ID: On Mon, 26 Sep 2022 15:35:28 GMT, Alan Bateman wrote: > BlockingChannelOps.java and BlockingSocketOps.java test virtual threads doing blocking I/O on channels and java.net sockets. > > BlockingChannelOps has 32 tests at this time and takes nearly 120s to run due to several tests that sleep to improve the chances that threads are blocked. These sleeps can be replaced with a poll of the thread state so the test runs in 3-4s. BlockingSocketOps has be changed to do the same time. > > In passing, I updated the tests in BlockingSocketOps that bound a ServerSocket to the wildcard address so they bind to the loopback address instead. This helps reduce potential interference in CI environments. I also put a workaround into BlockingChannelOps for macOS where the kernel appears to increase the amount of bytes that can be buffered in the socket sender buffer, it's otherwise too hard to test that socket writes block on that platform. This pull request has now been integrated. Changeset: 37f83b9b Author: Alan Bateman URL: https://git.openjdk.org/jdk/commit/37f83b9b8e007cd1153502540e9b1bf3cc713646 Stats: 610 lines in 2 files changed: 124 ins; 317 del; 169 mod 8294375: test/jdk/java/nio/channels/vthread/BlockingChannelOps.java is slow Reviewed-by: jpai ------------- PR: https://git.openjdk.org/jdk/pull/10427 From dfuchs at openjdk.org Wed Sep 28 13:24:26 2022 From: dfuchs at openjdk.org (Daniel Fuchs) Date: Wed, 28 Sep 2022 13:24:26 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni [v2] In-Reply-To: <1UZcv-X4dxQHlgj8b5B2sGJ9NpxWrOp1EwbWeazjj2I=.1cf2b14f-7064-47b6-8d25-887732020bb0@github.com> References: <3PJ-Gv4AW5nTD3gcQ1mwZH5C0jFo09zD6879B3jlkY4=.05beeb4b-498d-40fb-8514-244ac0ef0772@github.com> <1UZcv-X4dxQHlgj8b5B2sGJ9NpxWrOp1EwbWeazjj2I=.1cf2b14f-7064-47b6-8d25-887732020bb0@github.com> Message-ID: On Tue, 27 Sep 2022 23:12:43 GMT, Michael Ernst wrote: > Feel free to break up the pull request if that is what is needed to free it from red tape. Only you can do that @mernst ------------- PR: https://git.openjdk.org/jdk/pull/10029 From dfuchs at openjdk.org Wed Sep 28 13:38:37 2022 From: dfuchs at openjdk.org (Daniel Fuchs) Date: Wed, 28 Sep 2022 13:38:37 GMT Subject: RFR: 8294377: Prepare to stop auto-inheriting documentation for subclasses of exceptions whose documentation is inherited [v2] In-Reply-To: References: Message-ID: On Tue, 27 Sep 2022 12:14:23 GMT, Pavel Rappo wrote: >> This adds exception documentation to JDK methods that would otherwise lose that documentation once JDK-8287796 is integrated. While adding this exception documentation now does not change [^1] the JDK API Documentation, it will automatically counterbalance the effect that JDK-8287796 will have. >> >> JDK-8287796 seeks to remove an old, undocumented, and esoteric javadoc feature that cannot be suppressed [^2]. The feature is so little known about that IDEs either do not implement it correctly or do not implement it at all, thus rendering documentation differently from how the javadoc tool renders it. >> >> That feature was introduced in JDK-4947455 and manifests as follows. If a method inherits documentation for an exception, along with that documentation the method automatically inherits documentation for all exceptions that are subclasses of that former exception and are documented in an overridden method. >> >> To illustrate that behavior, consider the following example. A method `Subtype.m` inherits documentation for `SuperException`, while the overridden method `Supertype.m` documents `SuperException`, `SubExceptionA` and `SubExceptionB`, where the latter two exceptions are subclasses of the former exception: >> >> public class Subtype extends Supertype { >> >> @Override >> public void m() throws SuperException { >> ... >> >> public class Supertype { >> >> /** >> * @throws SuperException general problem >> * @throws SubExceptionA specific problem A >> * @throws SubExceptionB specific problem B >> */ >> public void m() throws SuperException { >> ... >> >> public class SuperException extends Exception { >> ... >> >> public class SubExceptionA extends SuperException { >> ... >> >> public class SubExceptionB extends SuperException { >> ... >> >> For the above example, the API Documentation for `Subtype.m` will contain documentation for all three exceptions; the page fragment for `Subtype.m` in "Method Details" will look like this: >> >> public void m() >> throws SuperException >> >> Overrides: >> m in class Supertype >> Throws: >> SuperException - general problem >> SubExceptionA - specific problem A >> SubExceptionB - specific problem B >> >> It works for checked and unchecked exceptions, for methods in classes and interfaces. >> >> >> If it's the first time you hear about that behavior and this change affects your area, it's a good opportunity to reflect on whether the exception documentation this change adds is really needed or you were simply unaware of the fact that it was automatically added before. If exception documentation is not needed, please comment on this PR. Otherwise, consider approving it. >> >> Keep in mind that if some exception documentation is not needed, **leaving it out** from this PR might require a CSR. >> >> >> [^1]: Aside from insignificant changes on class-use and index pages. There's one relatively significant change. This change is in jdk.jshell, where some methods disappeared from "Methods declared in ..." section and other irregularities. We are aware of this and have filed JDK-8291803 to fix the root cause. >> [^2]: In reality, the feature can be suppressed, but it looks like a bug rather than intentional design. If we legitimize the feature and its suppression behavior, the model for exception documentation inheritance might become much more complicated. > > Pavel Rappo has updated the pull request incrementally with one additional commit since the last revision: > > revert an extraneous change to jdk.javadoc The changes to JNDI and NIO look reasonable to me - though I haven't verified whether anything was missing (or no longer needed). If you can assert that the generated docs are the same before and after the proposed change then that's enough for me. src/java.naming/share/classes/javax/naming/directory/InitialDirContext.java line 186: > 184: /** > 185: * @throws AttributeModificationException {@inheritDoc} > 186: */ Isn't `{@inheritDoc}` needed in this case to preserve the method & params description? ------------- PR: https://git.openjdk.org/jdk/pull/10449 From prappo at openjdk.org Wed Sep 28 14:04:25 2022 From: prappo at openjdk.org (Pavel Rappo) Date: Wed, 28 Sep 2022 14:04:25 GMT Subject: RFR: 8294377: Prepare to stop auto-inheriting documentation for subclasses of exceptions whose documentation is inherited [v2] In-Reply-To: References: Message-ID: On Wed, 28 Sep 2022 13:34:08 GMT, Daniel Fuchs wrote: >> Pavel Rappo has updated the pull request incrementally with one additional commit since the last revision: >> >> revert an extraneous change to jdk.javadoc > > src/java.naming/share/classes/javax/naming/directory/InitialDirContext.java line 186: > >> 184: /** >> 185: * @throws AttributeModificationException {@inheritDoc} >> 186: */ > > Isn't `{@inheritDoc}` needed in this case to preserve the method & params description? Elements of a method declaration such as formal and type parameters as well as exceptions in the `throws` clause implicitly inherit descriptions, unless those elements are described in that method declaration. The same goes for the main description of a method: unless it exists in the method declaration, it is inherited. To sum up, those descriptions you ask about are implicitly inherited. No additional javadoc markup is needed. ------------- PR: https://git.openjdk.org/jdk/pull/10449 From alanb at openjdk.org Wed Sep 28 14:22:10 2022 From: alanb at openjdk.org (Alan Bateman) Date: Wed, 28 Sep 2022 14:22:10 GMT Subject: RFR: 8294377: Prepare to stop auto-inheriting documentation for subclasses of exceptions whose documentation is inherited [v2] In-Reply-To: References: Message-ID: On Tue, 27 Sep 2022 12:14:23 GMT, Pavel Rappo wrote: >> This adds exception documentation to JDK methods that would otherwise lose that documentation once JDK-8287796 is integrated. While adding this exception documentation now does not change [^1] the JDK API Documentation, it will automatically counterbalance the effect that JDK-8287796 will have. >> >> JDK-8287796 seeks to remove an old, undocumented, and esoteric javadoc feature that cannot be suppressed [^2]. The feature is so little known about that IDEs either do not implement it correctly or do not implement it at all, thus rendering documentation differently from how the javadoc tool renders it. >> >> That feature was introduced in JDK-4947455 and manifests as follows. If a method inherits documentation for an exception, along with that documentation the method automatically inherits documentation for all exceptions that are subclasses of that former exception and are documented in an overridden method. >> >> To illustrate that behavior, consider the following example. A method `Subtype.m` inherits documentation for `SuperException`, while the overridden method `Supertype.m` documents `SuperException`, `SubExceptionA` and `SubExceptionB`, where the latter two exceptions are subclasses of the former exception: >> >> public class Subtype extends Supertype { >> >> @Override >> public void m() throws SuperException { >> ... >> >> public class Supertype { >> >> /** >> * @throws SuperException general problem >> * @throws SubExceptionA specific problem A >> * @throws SubExceptionB specific problem B >> */ >> public void m() throws SuperException { >> ... >> >> public class SuperException extends Exception { >> ... >> >> public class SubExceptionA extends SuperException { >> ... >> >> public class SubExceptionB extends SuperException { >> ... >> >> For the above example, the API Documentation for `Subtype.m` will contain documentation for all three exceptions; the page fragment for `Subtype.m` in "Method Details" will look like this: >> >> public void m() >> throws SuperException >> >> Overrides: >> m in class Supertype >> Throws: >> SuperException - general problem >> SubExceptionA - specific problem A >> SubExceptionB - specific problem B >> >> It works for checked and unchecked exceptions, for methods in classes and interfaces. >> >> >> If it's the first time you hear about that behavior and this change affects your area, it's a good opportunity to reflect on whether the exception documentation this change adds is really needed or you were simply unaware of the fact that it was automatically added before. If exception documentation is not needed, please comment on this PR. Otherwise, consider approving it. >> >> Keep in mind that if some exception documentation is not needed, **leaving it out** from this PR might require a CSR. >> >> >> [^1]: Aside from insignificant changes on class-use and index pages. There's one relatively significant change. This change is in jdk.jshell, where some methods disappeared from "Methods declared in ..." section and other irregularities. We are aware of this and have filed JDK-8291803 to fix the root cause. >> [^2]: In reality, the feature can be suppressed, but it looks like a bug rather than intentional design. If we legitimize the feature and its suppression behavior, the model for exception documentation inheritance might become much more complicated. > > Pavel Rappo has updated the pull request incrementally with one additional commit since the last revision: > > revert an extraneous change to jdk.javadoc The updates to the channel classes look okay. ------------- PR: https://git.openjdk.org/jdk/pull/10449 From prappo at openjdk.org Wed Sep 28 14:22:11 2022 From: prappo at openjdk.org (Pavel Rappo) Date: Wed, 28 Sep 2022 14:22:11 GMT Subject: RFR: 8294377: Prepare to stop auto-inheriting documentation for subclasses of exceptions whose documentation is inherited [v2] In-Reply-To: References: Message-ID: On Wed, 28 Sep 2022 13:36:18 GMT, Daniel Fuchs wrote: > If you can assert that the generated docs are the same before and after the proposed change then that's enough for me. Would it suffice if I show the result of the diff between JDK API Documentation built at the tip of this PR branch (docs) and that of the jdk/master commit (docs-old) from which this PR branches? If so, please have a look: % diff -qnr build/macosx-aarch64/images/docs build/macosx-aarch64/images/docs-old Files build/macosx-aarch64/images/docs/api/index-files/index-9.html and build/macosx-aarch64/images/docs-old/api/index-files/index-9.html differ Files build/macosx-aarch64/images/docs/api/java.base/java/lang/class-use/Exception.html and build/macosx-aarch64/images/docs-old/api/java.base/java/lang/class-use/Exception.html differ Files build/macosx-aarch64/images/docs/api/java.base/java/lang/class-use/Override.html and build/macosx-aarch64/images/docs-old/api/java.base/java/lang/class-use/Override.html differ Files build/macosx-aarch64/images/docs/api/java.base/java/lang/class-use/String.html and build/macosx-aarch64/images/docs-old/api/java.base/java/lang/class-use/String.html differ Files build/macosx-aarch64/images/docs/api/java.base/java/lang/reflect/class-use/Method.html and build/macosx-aarch64/images/docs-old/api/java.base/java/lang/reflect/class-use/Method.html differ Files build/macosx-aarch64/images/docs/api/jdk.jshell/jdk/jshell/execution/JdiDefaultExecutionControl.html and build/macosx-aarch64/images/docs-old/api/jdk.jshell/jdk/jshell/execution/JdiDefaultExecutionControl.html differ Files build/macosx-aarch64/images/docs/api/jdk.jshell/jdk/jshell/execution/RemoteExecutionControl.html and build/macosx-aarch64/images/docs-old/api/jdk.jshell/jdk/jshell/execution/RemoteExecutionControl.html differ Files build/macosx-aarch64/images/docs/api/member-search-index.js and build/macosx-aarch64/images/docs-old/api/member-search-index.js differ As I said in the PR description, the differences are insignificant and confined to class-use and index structures. Well, except for jdk.jshell. But even there it might be okay for now. ------------- PR: https://git.openjdk.org/jdk/pull/10449 From mernst at openjdk.org Wed Sep 28 14:49:20 2022 From: mernst at openjdk.org (Michael Ernst) Date: Wed, 28 Sep 2022 14:49:20 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni [v2] In-Reply-To: References: Message-ID: On Mon, 26 Sep 2022 16:51:36 GMT, Michael Ernst wrote: >> 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni > > Michael Ernst has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits: > > - Reinstate typos in Apache code that is copied into the JDK > - Merge ../jdk-openjdk into typos-typos > - Remove file that was removed upstream > - Fix inconsistency in capitalization > - Undo change in zlip > - Fix typos Regarding breaking up the pull request: * No one has stated that this is actually necessary or given a guarantee that it will result in a merged pull request, only that it *might* finally free this pull request of red tape. * I have spent a nontrivial amount of time on these fixes. There have been long delays, which forced me to resolve merge conflicts. I was told that the changes are "mostly fine" without any indication of what is wrong. I am reluctant to spend yet more time, only to later (maybe weeks later, as has already happened) find out that I still have to do something else or that all my effort was wasted. * No one has explained what to do. "separate PRs by team" doesn't tell me what to do, since I don't see an org chart anywhere in the documentation that tells me what a "team" is. I'm sorry to state it so bluntly, but this is not a good look for the JDK team being welcoming to corrections. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From dfuchs at openjdk.org Wed Sep 28 15:13:39 2022 From: dfuchs at openjdk.org (Daniel Fuchs) Date: Wed, 28 Sep 2022 15:13:39 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni [v2] In-Reply-To: References: Message-ID: On Wed, 28 Sep 2022 14:45:54 GMT, Michael Ernst wrote: >> Michael Ernst has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits: >> >> - Reinstate typos in Apache code that is copied into the JDK >> - Merge ../jdk-openjdk into typos-typos >> - Remove file that was removed upstream >> - Fix inconsistency in capitalization >> - Undo change in zlip >> - Fix typos > > Regarding breaking up the pull request: > * No one has stated that this is actually necessary or given a guarantee that it will result in a merged pull request, only that it *might* finally free this pull request of red tape. > * Despite my repeated requests, no one has justified that *this particular pull request* needs the expertise of different teams. The only reasons stated have been custom and bureaucracy. > * I have spent a nontrivial amount of time on these fixes. There have been long delays, which forced me to resolve merge conflicts. I was told that the changes are "mostly fine" without any indication of what is wrong. I am reluctant to spend yet more time, only to later (maybe weeks later, as has already happened) find out that I still have to do something else or that all my effort was wasted. > * I was condescendingly told "We have discussed several times in the past ..." even though I was not privy to those discussions. No pointer to documentation in the JDK itself, about pull requests, was given. > * No one has explained what to do. "separate PRs by team" doesn't tell me what to do, since I don't see an org chart anywhere in the documentation that tells me what a "team" is. > > I'm sorry to state it so bluntly, but this is not a good look for the JDK team being welcoming to contributions. Hi @mernst, sorry if this is how it appears. Pull requests that involve changes spanning a large set of files are notoriously harder to review since they require a reviewer from each different area to chime in. Basically you will need one reviewer from each of these areas: - client - compiler - core-libs - hotspot - i18n - javadoc - jmx - net - nio - security - serviceability Jaikiran suggested to reduce the scope of this PR in his first comment: https://github.com/openjdk/jdk/pull/10029#issuecomment-1236588632 as did several other reviewers. Sorry if the meaning (or reason) was not clear. best regards, ------------- PR: https://git.openjdk.org/jdk/pull/10029 From aivanov at openjdk.org Wed Sep 28 15:28:24 2022 From: aivanov at openjdk.org (Alexey Ivanov) Date: Wed, 28 Sep 2022 15:28:24 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni [v2] In-Reply-To: References: Message-ID: On Mon, 26 Sep 2022 16:51:36 GMT, Michael Ernst wrote: >> 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni > > Michael Ernst has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits: > > - Reinstate typos in Apache code that is copied into the JDK > - Merge ../jdk-openjdk into typos-typos > - Remove file that was removed upstream > - Fix inconsistency in capitalization > - Undo change in zlip > - Fix typos The title says under `test/jdk/*`, yet there are lots of files changed in `src/*`. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From mernst at openjdk.org Wed Sep 28 15:35:40 2022 From: mernst at openjdk.org (Michael Ernst) Date: Wed, 28 Sep 2022 15:35:40 GMT Subject: RFR: 8294321: Fix typos in files under test/jdk/java, test/jdk/jdk, test/jdk/jni [v2] In-Reply-To: References: Message-ID: On Wed, 28 Sep 2022 15:26:13 GMT, Alexey Ivanov wrote: > The title says under `test/jdk/*`, yet there are lots of files changed in `src/*`. The title was edited by someone other than me, as you can see from the PR history. ------------- PR: https://git.openjdk.org/jdk/pull/10029 From alanb at openjdk.org Wed Sep 28 19:07:23 2022 From: alanb at openjdk.org (Alan Bateman) Date: Wed, 28 Sep 2022 19:07:23 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows In-Reply-To: References: Message-ID: <3OW_XmA2gpEHpcjAQ--6E4OdLGdochUhw-MyRHSHq2E=.2ad0f6e2-6013-4bce-b155-86a96452a91a@github.com> On Wed, 28 Sep 2022 10:58:43 GMT, Daniel Jeli?ski wrote: > Please review this test-only change that improves the execution speed of a few FileChannel tests: > - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. > - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. > - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) Creating the files as sparse files make sense, we probably should have changed these tests a long time ago. test/jdk/java/nio/channels/FileChannel/LargeMapTest.java line 64: > 62: try (FileChannel fc = FileChannel.open(p, CREATE_NEW, SPARSE, WRITE)) { > 63: fc.position(BASE); > 64: Random r = new Random(System.nanoTime()); It might not matter here but we usually use RandomFactory to allow for reproducibility in the event of test failure. test/jdk/java/nio/channels/FileChannel/MapTest.java line 95: > 93: t1 = System.nanoTime(); > 94: out.printf("Exceptions: done in %d ns (%d ms) %n", > 95: t1 - t0, TimeUnit.NANOSECONDS.toMillis(t1 - t0)); Are you planning to leave all these timestamps in the output? I can't tell if they are left over from your testing or intentional. test/jdk/java/nio/channels/FileChannel/Transfer2GPlus.java line 81: > 79: throws IOException { > 80: try (FileChannel fc = FileChannel.open(src, StandardOpenOption.CREATE_NEW, > 81: StandardOpenOption.SPARSE, StandardOpenOption.WRITE)) { LargeMapFile uses import static java.nio.file.StandardOpenOption.* which makes it easier to fit the open options on the same line. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From djelinski at openjdk.org Wed Sep 28 19:12:20 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Wed, 28 Sep 2022 19:12:20 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows In-Reply-To: <3OW_XmA2gpEHpcjAQ--6E4OdLGdochUhw-MyRHSHq2E=.2ad0f6e2-6013-4bce-b155-86a96452a91a@github.com> References: <3OW_XmA2gpEHpcjAQ--6E4OdLGdochUhw-MyRHSHq2E=.2ad0f6e2-6013-4bce-b155-86a96452a91a@github.com> Message-ID: On Wed, 28 Sep 2022 19:01:45 GMT, Alan Bateman wrote: >> Please review this test-only change that improves the execution speed of a few FileChannel tests: >> - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. >> - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. >> - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) > > test/jdk/java/nio/channels/FileChannel/MapTest.java line 95: > >> 93: t1 = System.nanoTime(); >> 94: out.printf("Exceptions: done in %d ns (%d ms) %n", >> 95: t1 - t0, TimeUnit.NANOSECONDS.toMillis(t1 - t0)); > > Are you planning to leave all these timestamps in the output? I can't tell if they are left over from your testing or intentional. Yeah, I wanted to leave them here; they don't take up much space, and are very useful when trying to figure out why the tests are slow. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From djelinski at openjdk.org Wed Sep 28 19:26:23 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Wed, 28 Sep 2022 19:26:23 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows In-Reply-To: <3OW_XmA2gpEHpcjAQ--6E4OdLGdochUhw-MyRHSHq2E=.2ad0f6e2-6013-4bce-b155-86a96452a91a@github.com> References: <3OW_XmA2gpEHpcjAQ--6E4OdLGdochUhw-MyRHSHq2E=.2ad0f6e2-6013-4bce-b155-86a96452a91a@github.com> Message-ID: <_cFQm5B1RZtYVGufBp44MyzSJZf4XlW4Wjzh_GwZziM=.cbb5c0b5-c072-41d2-bd51-99579c4f208d@github.com> On Wed, 28 Sep 2022 19:01:14 GMT, Alan Bateman wrote: >> Please review this test-only change that improves the execution speed of a few FileChannel tests: >> - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. >> - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. >> - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) > > test/jdk/java/nio/channels/FileChannel/LargeMapTest.java line 64: > >> 62: try (FileChannel fc = FileChannel.open(p, CREATE_NEW, SPARSE, WRITE)) { >> 63: fc.position(BASE); >> 64: Random r = new Random(System.nanoTime()); > > It might not matter here but we usually use RandomFactory to allow for reproducibility in the event of test failure. Thanks for the hint! It shouldn't matter here, but I'll change it. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From bpb at openjdk.org Wed Sep 28 19:53:21 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Wed, 28 Sep 2022 19:53:21 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows In-Reply-To: References: Message-ID: On Wed, 28 Sep 2022 10:58:43 GMT, Daniel Jeli?ski wrote: > Please review this test-only change that improves the execution speed of a few FileChannel tests: > - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. > - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. > - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) Overall I don?t see any problems aside from the copyright years which I assume you will fix before integrating. It?s good that you removed `RandomAccessFile` in one place. It might be good to try to remove all `java.io` uses where possible, e.g., not `deleteOnExit()`. Also, instead of creating a temporary file and then deleting it, maybe something else could be done. For example, creating a temporary directory and then creating a file with a hard-coded name within it. Or using `Random` to create a file name and using the `REPLACE_EXISTING` open option. Another thought would be to add a method `createSparseTempFile()` or something to `lib/util/FileUtils`. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From bpb at openjdk.org Thu Sep 29 01:26:37 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 29 Sep 2022 01:26:37 GMT Subject: RFR: 8294519: (fs) java/nio/file/Files/CopyProcFile.java fails intermittenly due to unstable /proc/cpuinfo output Message-ID: 1. Change source from `/proc/cpuinfo` to `/proc/version`. 2. Do not swallow errors in `@BeforeTest` config action. 3. Remove test from problem list. ------------- Commit messages: - 8294519: (fs) java/nio/file/Files/CopyProcFile.java fails intermittenly due to unstable /proc/cpuinfo output Changes: https://git.openjdk.org/jdk/pull/10479/files Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=10479&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8294519 Stats: 13 lines in 2 files changed: 3 ins; 6 del; 4 mod Patch: https://git.openjdk.org/jdk/pull/10479.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10479/head:pull/10479 PR: https://git.openjdk.org/jdk/pull/10479 From alanb at openjdk.org Thu Sep 29 07:14:16 2022 From: alanb at openjdk.org (Alan Bateman) Date: Thu, 29 Sep 2022 07:14:16 GMT Subject: RFR: 8294519: (fs) java/nio/file/Files/CopyProcFile.java fails intermittenly due to unstable /proc/cpuinfo output In-Reply-To: References: Message-ID: <9gsM-P0EX7wgtX3IQMtZSheL2-NlCP9YqCqRJ66kDs8=.97880171-d74d-4d7a-a21c-679bfb5659c5@github.com> On Thu, 29 Sep 2022 01:17:36 GMT, Brian Burkhalter wrote: > 1. Change source from `/proc/cpuinfo` to `/proc/version`. > 2. Do not swallow errors in `@BeforeTest` config action. > 3. Remove test from problem list. file /proc/version reports that it is empty but you can read content, so I think it works to exercise the code paths. ------------- Marked as reviewed by alanb (Reviewer). PR: https://git.openjdk.org/jdk/pull/10479 From shade at openjdk.org Thu Sep 29 09:35:48 2022 From: shade at openjdk.org (Aleksey Shipilev) Date: Thu, 29 Sep 2022 09:35:48 GMT Subject: RFR: 8294519: (fs) java/nio/file/Files/CopyProcFile.java fails intermittenly due to unstable /proc/cpuinfo output In-Reply-To: References: Message-ID: On Thu, 29 Sep 2022 01:17:36 GMT, Brian Burkhalter wrote: > 1. Change source from `/proc/cpuinfo` to `/proc/version`. > 2. Do not swallow errors in `@BeforeTest` config action. > 3. Remove test from problem list. Looks okay to me, and test passes on the system where it failed before. test/jdk/java/nio/file/Files/CopyProcFile.java line 127: > 125: throw e; > 126: } > 127: if (Files.mismatch(Path.of(BUFFERED_COPY), Path.of(SOURCE)) != -1) Braces? ------------- Marked as reviewed by shade (Reviewer). PR: https://git.openjdk.org/jdk/pull/10479 From lancea at openjdk.org Thu Sep 29 10:36:20 2022 From: lancea at openjdk.org (Lance Andersen) Date: Thu, 29 Sep 2022 10:36:20 GMT Subject: RFR: 8294519: (fs) java/nio/file/Files/CopyProcFile.java fails intermittenly due to unstable /proc/cpuinfo output In-Reply-To: References: Message-ID: On Thu, 29 Sep 2022 01:17:36 GMT, Brian Burkhalter wrote: > 1. Change source from `/proc/cpuinfo` to `/proc/version`. > 2. Do not swallow errors in `@BeforeTest` config action. > 3. Remove test from problem list. Marked as reviewed by lancea (Reviewer). ------------- PR: https://git.openjdk.org/jdk/pull/10479 From djelinski at openjdk.org Thu Sep 29 14:30:21 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Thu, 29 Sep 2022 14:30:21 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v2] In-Reply-To: References: Message-ID: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> > Please review this test-only change that improves the execution speed of a few FileChannel tests: > - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. > - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. > - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) Daniel Jeli?ski has updated the pull request incrementally with seven additional commits since the last revision: - Remove intermittently failing deletes - reintroduce deleteOnExit - rewrite testToWritableByteChannel without using file output - Migrate Transfer4GBFile to nio - Implement createSparseTempFile - Use RandomFactory to generate random bytes - Update copyright year ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10464/files - new: https://git.openjdk.org/jdk/pull/10464/files/51b50f48..614d0e54 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10464&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10464&range=00-01 Stats: 188 lines in 6 files changed: 62 ins; 50 del; 76 mod Patch: https://git.openjdk.org/jdk/pull/10464.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10464/head:pull/10464 PR: https://git.openjdk.org/jdk/pull/10464 From djelinski at openjdk.org Thu Sep 29 14:51:22 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Thu, 29 Sep 2022 14:51:22 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v2] In-Reply-To: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> References: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> Message-ID: On Thu, 29 Sep 2022 14:30:21 GMT, Daniel Jeli?ski wrote: >> Please review this test-only change that improves the execution speed of a few FileChannel tests: >> - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. >> - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. >> - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) > > Daniel Jeli?ski has updated the pull request incrementally with seven additional commits since the last revision: > > - Remove intermittently failing deletes > - reintroduce deleteOnExit > - rewrite testToWritableByteChannel without using file output > - Migrate Transfer4GBFile to nio > - Implement createSparseTempFile > - Use RandomFactory to generate random bytes > - Update copyright year I went with adding `FileUtils.createSparseTempFile()`; the method implementation is still ugly, but at least it's not visible when looking at the tests. I added one more modification to `Transfer2GPlus`; noticed that `testToWritableByteChannel` doesn't need to write to disk, and modified the code to take advantage of that; it's much faster now. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From alanb at openjdk.org Thu Sep 29 15:08:19 2022 From: alanb at openjdk.org (Alan Bateman) Date: Thu, 29 Sep 2022 15:08:19 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v2] In-Reply-To: References: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> Message-ID: <9BGjUDNZ0TN-i4L4WKm_HsPXcGVJAP0hnJIw5qLKJDI=.6caaa8ee-9407-4240-8171-bea5b197a336@github.com> On Thu, 29 Sep 2022 14:47:55 GMT, Daniel Jeli?ski wrote: > I went with adding `FileUtils.createSparseTempFile()`; the method implementation is still ugly, but at least it's not visible when looking at the tests. I would prefer if the FileChannel tests do not use this. The reason is that these are tests for FileChannel and should not be dependent on test infrastructure that it out of sight. It would be okay for non-FileChannel tests to use of course. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From djelinski at openjdk.org Thu Sep 29 15:15:24 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Thu, 29 Sep 2022 15:15:24 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v2] In-Reply-To: <9BGjUDNZ0TN-i4L4WKm_HsPXcGVJAP0hnJIw5qLKJDI=.6caaa8ee-9407-4240-8171-bea5b197a336@github.com> References: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> <9BGjUDNZ0TN-i4L4WKm_HsPXcGVJAP0hnJIw5qLKJDI=.6caaa8ee-9407-4240-8171-bea5b197a336@github.com> Message-ID: On Thu, 29 Sep 2022 15:06:14 GMT, Alan Bateman wrote: > ... should not be dependent on test infrastructure that it out of sight. would it be okay if I moved the method to, say, FileChannelUtils in FileChannel directory? ------------- PR: https://git.openjdk.org/jdk/pull/10464 From bpb at openjdk.org Thu Sep 29 18:16:37 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 29 Sep 2022 18:16:37 GMT Subject: RFR: 8294519: (fs) java/nio/file/Files/CopyProcFile.java fails intermittenly due to unstable /proc/cpuinfo output [v2] In-Reply-To: References: Message-ID: <1kdIvqsdoJAyZHObFhwsAK8gXfoYQZiwNHJ3Gxx2_C8=.9ace0ef4-d2fa-4f93-97f2-9efd34d42b00@github.com> > 1. Change source from `/proc/cpuinfo` to `/proc/version`. > 2. Do not swallow errors in `@BeforeTest` config action. > 3. Remove test from problem list. Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: 8294519: Add some braces for better readability ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10479/files - new: https://git.openjdk.org/jdk/pull/10479/files/c80329e6..680ad7de Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10479&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10479&range=00-01 Stats: 6 lines in 1 file changed: 3 ins; 0 del; 3 mod Patch: https://git.openjdk.org/jdk/pull/10479.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10479/head:pull/10479 PR: https://git.openjdk.org/jdk/pull/10479 From lancea at openjdk.org Thu Sep 29 20:46:33 2022 From: lancea at openjdk.org (Lance Andersen) Date: Thu, 29 Sep 2022 20:46:33 GMT Subject: RFR: 8294519: (fs) java/nio/file/Files/CopyProcFile.java fails intermittenly due to unstable /proc/cpuinfo output [v2] In-Reply-To: <1kdIvqsdoJAyZHObFhwsAK8gXfoYQZiwNHJ3Gxx2_C8=.9ace0ef4-d2fa-4f93-97f2-9efd34d42b00@github.com> References: <1kdIvqsdoJAyZHObFhwsAK8gXfoYQZiwNHJ3Gxx2_C8=.9ace0ef4-d2fa-4f93-97f2-9efd34d42b00@github.com> Message-ID: On Thu, 29 Sep 2022 18:16:37 GMT, Brian Burkhalter wrote: >> 1. Change source from `/proc/cpuinfo` to `/proc/version`. >> 2. Do not swallow errors in `@BeforeTest` config action. >> 3. Remove test from problem list. > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8294519: Add some braces for better readability Marked as reviewed by lancea (Reviewer). ------------- PR: https://git.openjdk.org/jdk/pull/10479 From bpb at openjdk.org Thu Sep 29 21:09:28 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 29 Sep 2022 21:09:28 GMT Subject: Integrated: 8294519: (fs) java/nio/file/Files/CopyProcFile.java fails intermittenly due to unstable /proc/cpuinfo output In-Reply-To: References: Message-ID: <3TwR-773s-wlfYURI5fzmqR846ao9yBrESJSbCxy0p8=.9d630dec-1373-4d3b-8e30-3299a52e4630@github.com> On Thu, 29 Sep 2022 01:17:36 GMT, Brian Burkhalter wrote: > 1. Change source from `/proc/cpuinfo` to `/proc/version`. > 2. Do not swallow errors in `@BeforeTest` config action. > 3. Remove test from problem list. This pull request has now been integrated. Changeset: a07975bf Author: Brian Burkhalter URL: https://git.openjdk.org/jdk/commit/a07975bf3e321bdf02748693930c2523fa1df22d Stats: 18 lines in 2 files changed: 6 ins; 6 del; 6 mod 8294519: (fs) java/nio/file/Files/CopyProcFile.java fails intermittenly due to unstable /proc/cpuinfo output Reviewed-by: alanb, shade, lancea ------------- PR: https://git.openjdk.org/jdk/pull/10479 From bpb at openjdk.org Thu Sep 29 23:54:16 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Thu, 29 Sep 2022 23:54:16 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v2] In-Reply-To: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> References: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> Message-ID: On Thu, 29 Sep 2022 14:30:21 GMT, Daniel Jeli?ski wrote: >> Please review this test-only change that improves the execution speed of a few FileChannel tests: >> - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. >> - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. >> - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) > > Daniel Jeli?ski has updated the pull request incrementally with seven additional commits since the last revision: > > - Remove intermittently failing deletes > - reintroduce deleteOnExit > - rewrite testToWritableByteChannel without using file output > - Migrate Transfer4GBFile to nio > - Implement createSparseTempFile > - Use RandomFactory to generate random bytes > - Update copyright year test/jdk/java/nio/channels/FileChannel/LargeMapTest.java line 47: > 45: * EXCEPTION_ACCESS_VIOLATION. > 46: * @requires vm.bits == 64 > 47: * @library /test/lib Typically tests which use randomly generated values also have the tag * @key randomness ------------- PR: https://git.openjdk.org/jdk/pull/10464 From bpb at openjdk.org Fri Sep 30 00:01:20 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 30 Sep 2022 00:01:20 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v2] In-Reply-To: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> References: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> Message-ID: On Thu, 29 Sep 2022 14:30:21 GMT, Daniel Jeli?ski wrote: >> Please review this test-only change that improves the execution speed of a few FileChannel tests: >> - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. >> - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. >> - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) > > Daniel Jeli?ski has updated the pull request incrementally with seven additional commits since the last revision: > > - Remove intermittently failing deletes > - reintroduce deleteOnExit > - rewrite testToWritableByteChannel without using file output > - Migrate Transfer4GBFile to nio > - Implement createSparseTempFile > - Use RandomFactory to generate random bytes > - Update copyright year test/jdk/java/nio/channels/FileChannel/Transfer2GPlus.java line 140: > 138: > 139: long n; > 140: if ((n = srcCh.transferTo(0, LENGTH, wbc)) < LENGTH) The contract of `FileChannel::transferTo` does not guarantee that all bytes are transferred: ```An invocation of this method may or may not transfer all of the requested bytes [...]```. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From bpb at openjdk.org Fri Sep 30 00:08:19 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 30 Sep 2022 00:08:19 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v2] In-Reply-To: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> References: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> Message-ID: On Thu, 29 Sep 2022 14:30:21 GMT, Daniel Jeli?ski wrote: >> Please review this test-only change that improves the execution speed of a few FileChannel tests: >> - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. >> - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. >> - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) > > Daniel Jeli?ski has updated the pull request incrementally with seven additional commits since the last revision: > > - Remove intermittently failing deletes > - reintroduce deleteOnExit > - rewrite testToWritableByteChannel without using file output > - Migrate Transfer4GBFile to nio > - Implement createSparseTempFile > - Use RandomFactory to generate random bytes > - Update copyright year test/jdk/java/nio/channels/FileChannel/Transfer4GBFile.java line 73: > 71: StandardOpenOption.READ); > 72: FileChannel sinkChannel = FileChannel.open(sink, > 73: StandardOpenOption.WRITE)) { Some weird indentation at lines 71 and 73. A static import of the `StandardOpenOption` enum onstants would help. test/jdk/java/nio/channels/FileChannel/Transfer4GBFile.java line 116: > 114: try (FileChannel sourceChannel = FileChannel.open(source, StandardOpenOption.READ); > 115: FileChannel sinkChannel = FileChannel.open(sink, StandardOpenOption.WRITE)) { > 116: long bytesWritten = sinkChannel.transferFrom(sourceChannel, Same comment as for `FileChannel::transferTo`: ```An invocation of this method may or may not transfer all of the requested bytes```. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From mkartashev at openjdk.org Fri Sep 30 05:17:43 2022 From: mkartashev at openjdk.org (Maxim Kartashev) Date: Fri, 30 Sep 2022 05:17:43 GMT Subject: RFR: 8293067: (fs) Implement WatchService using system library (macOS) [v7] In-Reply-To: References: Message-ID: > This is an implementation of `WatchService` based on File System Events API that is capable of generating events whenever a change occurs in an interesting directory or underneath it. Since the API naturally supports "recursive" watch, the `FILE_TREE` is supported by the watch service. > > Some things of note: > * There's one "service" thread per `WatchService` instance that is inactive unless changes occur in the watched directory. The changes are grouped by introducing a time delay between when they occurred and when they are reported, which is controlled by the sensitivity modifier of the watch service. > * Since FSEvents API reports directories only, the watch service keeps a snapshot (hierarchical if necessary) of the files in the directory being watched. The snapshot gets updated when an event in that directory or underneath it gets delivered. File changes are detected by comparing "last modified" time with a millisecond precision (`BasicFileAttributes.lastModifiedTime()`). > * There is a slight complication with the move of an entire directory hierarchy: FSEvents API only reports about the containing directory of that move and not about any of the directories actually moved. There's a separate test for that (`Move.java`). > * The code is careful not to do any I/O (such as reading the contents of a directory or attributes of a file) unless unavoidable. Any deviation from this line should be considered a bug (of, arguably, low priority). > * The native part consists mostly of API wrappers with one exception of the callback function invoked by the system to report the events that occurred. The sole task of the function is to convert from C strings to Java strings and pass the array of affected directories to Java code. This can be rewritten if desired to make the code more future-proof. > > This commit leaves `PollingWatchService` unused. I'm not sure if I should/can do anything about it. Any advice is welcomed. > > ### Testing > > * Tested by running `test/jdk/java/nio/file` and `test/jdk/jdk/nio` on MacOS 10.15.7 (x64) and `test/jdk/java/nio/` plus `test/jdk/jdk/nio` on MacOS 12.5.1 (aarch64). > * Also verified that new tests pass on Linux and Windows. > * This code (albeit in a slightly modified form) has been in use at JetBrains for around half a year and a few bugs have been found and fixed during that time period. Maxim Kartashev has updated the pull request incrementally with one additional commit since the last revision: Dropped __unused attributes from used parameters ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10140/files - new: https://git.openjdk.org/jdk/pull/10140/files/d834262f..8d5c1962 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=06 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10140&range=05-06 Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod Patch: https://git.openjdk.org/jdk/pull/10140.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10140/head:pull/10140 PR: https://git.openjdk.org/jdk/pull/10140 From djelinski at openjdk.org Fri Sep 30 06:07:32 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Fri, 30 Sep 2022 06:07:32 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v2] In-Reply-To: References: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> Message-ID: <-5rjpuhDz69l4H16AXbDNLic9jo9qmZgD8O7oJSdT_s=.8c48d614-a6a3-4fd8-b0aa-a0040b2d07ef@github.com> On Thu, 29 Sep 2022 23:59:04 GMT, Brian Burkhalter wrote: >> Daniel Jeli?ski has updated the pull request incrementally with seven additional commits since the last revision: >> >> - Remove intermittently failing deletes >> - reintroduce deleteOnExit >> - rewrite testToWritableByteChannel without using file output >> - Migrate Transfer4GBFile to nio >> - Implement createSparseTempFile >> - Use RandomFactory to generate random bytes >> - Update copyright year > > test/jdk/java/nio/channels/FileChannel/Transfer2GPlus.java line 140: > >> 138: >> 139: long n; >> 140: if ((n = srcCh.transferTo(0, LENGTH, wbc)) < LENGTH) > > The contract of `FileChannel::transferTo` does not guarantee that all bytes are transferred: > > ```An invocation of this method may or may not transfer all of the requested bytes [...]```. preexisting, and unlikely to change because of Hyrum's law. I can file a ticket to update the docs if you think it makes sense. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From djelinski at openjdk.org Fri Sep 30 10:36:54 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Fri, 30 Sep 2022 10:36:54 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v3] In-Reply-To: References: Message-ID: <5F-Y8ZWbYyBKFhiXs1-ScQwHoMMVzylBqtqcYYuXZkA=.16075689-490a-43a8-a50f-9c05923eb38a@github.com> > Please review this test-only change that improves the execution speed of a few FileChannel tests: > - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. > - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. > - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) Daniel Jeli?ski has updated the pull request incrementally with four additional commits since the last revision: - Add explicit build tags - Move createSparseTempFile closer to FileChannel - add randomness - Import static StandardOpenOption ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10464/files - new: https://git.openjdk.org/jdk/pull/10464/files/614d0e54..08829f02 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10464&range=02 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10464&range=01-02 Stats: 96 lines in 6 files changed: 52 ins; 25 del; 19 mod Patch: https://git.openjdk.org/jdk/pull/10464.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10464/head:pull/10464 PR: https://git.openjdk.org/jdk/pull/10464 From djelinski at openjdk.org Fri Sep 30 10:53:08 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Fri, 30 Sep 2022 10:53:08 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v4] In-Reply-To: References: Message-ID: > Please review this test-only change that improves the execution speed of a few FileChannel tests: > - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. > - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. > - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) Daniel Jeli?ski has updated the pull request incrementally with one additional commit since the last revision: Fix file attributes ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10464/files - new: https://git.openjdk.org/jdk/pull/10464/files/08829f02..a8a28f1d Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10464&range=03 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10464&range=02-03 Stats: 0 lines in 1 file changed: 0 ins; 0 del; 0 mod Patch: https://git.openjdk.org/jdk/pull/10464.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10464/head:pull/10464 PR: https://git.openjdk.org/jdk/pull/10464 From alanb at openjdk.org Fri Sep 30 11:15:30 2022 From: alanb at openjdk.org (Alan Bateman) Date: Fri, 30 Sep 2022 11:15:30 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v4] In-Reply-To: References: Message-ID: <6L0hiaFEkXLvtkxvx7dIGxO9KiDaUW3PuRxvepG02tw=.dbda303c-69f3-4a8f-9548-caadfba2b49f@github.com> On Fri, 30 Sep 2022 10:53:08 GMT, Daniel Jeli?ski wrote: >> Please review this test-only change that improves the execution speed of a few FileChannel tests: >> - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. >> - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. >> - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) > > Daniel Jeli?ski has updated the pull request incrementally with one additional commit since the last revision: > > Fix file attributes I think this version looks okay and probably okay drop the loop from testForce. It's probably best to wait until @bplb gets to see the updated version as he has been doing most of the work in this area recently. test/jdk/java/nio/channels/FileChannel/TransferTo6GBFile.java line 63: > 61: out.println(" Writing large file..."); > 62: long t0 = System.nanoTime(); > 63: FileChannel fc = FileChannel.open(file, READ, WRITE); You can probably change this to use try-with-resources too and maybe "READ" can be dropped as this loo just writes to the file. ------------- Marked as reviewed by alanb (Reviewer). PR: https://git.openjdk.org/jdk/pull/10464 From bpb at openjdk.org Fri Sep 30 15:03:25 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 30 Sep 2022 15:03:25 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v2] In-Reply-To: <-5rjpuhDz69l4H16AXbDNLic9jo9qmZgD8O7oJSdT_s=.8c48d614-a6a3-4fd8-b0aa-a0040b2d07ef@github.com> References: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> <-5rjpuhDz69l4H16AXbDNLic9jo9qmZgD8O7oJSdT_s=.8c48d614-a6a3-4fd8-b0aa-a0040b2d07ef@github.com> Message-ID: On Fri, 30 Sep 2022 06:03:13 GMT, Daniel Jeli?ski wrote: >> test/jdk/java/nio/channels/FileChannel/Transfer2GPlus.java line 140: >> >>> 138: >>> 139: long n; >>> 140: if ((n = srcCh.transferTo(0, LENGTH, wbc)) < LENGTH) >> >> The contract of `FileChannel::transferTo` does not guarantee that all bytes are transferred: >> >> ```An invocation of this method may or may not transfer all of the requested bytes [...]```. > > preexisting, and unlikely to change because of Hyrum's law. I can file a ticket to update the docs if you think it makes sense. No, I think the docs should be left as they are. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From bpb at openjdk.org Fri Sep 30 16:30:37 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 30 Sep 2022 16:30:37 GMT Subject: RFR: 8057113: (fs) Path should have a method to obtain the filename extension [v18] In-Reply-To: References: Message-ID: On Mon, 1 Aug 2022 21:54:22 GMT, Brian Burkhalter wrote: >> Resurrection of the proposal to add a method to obtain the filename extension originated in PR [2319](https://github.com/openjdk/jdk/pull/2319). > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8057113: Remove API note `continue;` ------------- PR: https://git.openjdk.org/jdk/pull/8066 From bpb at openjdk.org Fri Sep 30 16:36:38 2022 From: bpb at openjdk.org (Brian Burkhalter) Date: Fri, 30 Sep 2022 16:36:38 GMT Subject: RFR: 8057113: (fs) Path should have a method to obtain the filename extension [v18] In-Reply-To: References: Message-ID: <0SB-fa1FU9PF5_OhxSJAzmFPR8i6Tnfjm04YjRr_fb4=.46c5763f-fb08-40ad-9c26-ae6d4f732884@github.com> On Mon, 1 Aug 2022 21:54:22 GMT, Brian Burkhalter wrote: >> Resurrection of the proposal to add a method to obtain the filename extension originated in PR [2319](https://github.com/openjdk/jdk/pull/2319). > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8057113: Remove API note The principal unresolved issue here concerns leading period characters. There is consensus that the extension of a file name string such as `.foo` is `null`, but not as to whether the extension of a file name string such as `...foo` is `null` or `foo`. My intent was that if there were multiple leading periods, then all but the last would be treated as if they were alphabetic, hence the behavior for `...foo` would be the same as for `xy.foo`. I could be persuaded otherwise, and the alternative is likely simpler. ------------- PR: https://git.openjdk.org/jdk/pull/8066 From prr at openjdk.org Fri Sep 30 19:17:22 2022 From: prr at openjdk.org (Phil Race) Date: Fri, 30 Sep 2022 19:17:22 GMT Subject: RFR: 8294377: Prepare to stop auto-inheriting documentation for subclasses of exceptions whose documentation is inherited [v2] In-Reply-To: References: Message-ID: On Tue, 27 Sep 2022 12:14:23 GMT, Pavel Rappo wrote: >> This adds exception documentation to JDK methods that would otherwise lose that documentation once JDK-8287796 is integrated. While adding this exception documentation now does not change [^1] the JDK API Documentation, it will automatically counterbalance the effect that JDK-8287796 will have. >> >> JDK-8287796 seeks to remove an old, undocumented, and esoteric javadoc feature that cannot be suppressed [^2]. The feature is so little known about that IDEs either do not implement it correctly or do not implement it at all, thus rendering documentation differently from how the javadoc tool renders it. >> >> That feature was introduced in JDK-4947455 and manifests as follows. If a method inherits documentation for an exception, along with that documentation the method automatically inherits documentation for all exceptions that are subclasses of that former exception and are documented in an overridden method. >> >> To illustrate that behavior, consider the following example. A method `Subtype.m` inherits documentation for `SuperException`, while the overridden method `Supertype.m` documents `SuperException`, `SubExceptionA` and `SubExceptionB`, where the latter two exceptions are subclasses of the former exception: >> >> public class Subtype extends Supertype { >> >> @Override >> public void m() throws SuperException { >> ... >> >> public class Supertype { >> >> /** >> * @throws SuperException general problem >> * @throws SubExceptionA specific problem A >> * @throws SubExceptionB specific problem B >> */ >> public void m() throws SuperException { >> ... >> >> public class SuperException extends Exception { >> ... >> >> public class SubExceptionA extends SuperException { >> ... >> >> public class SubExceptionB extends SuperException { >> ... >> >> For the above example, the API Documentation for `Subtype.m` will contain documentation for all three exceptions; the page fragment for `Subtype.m` in "Method Details" will look like this: >> >> public void m() >> throws SuperException >> >> Overrides: >> m in class Supertype >> Throws: >> SuperException - general problem >> SubExceptionA - specific problem A >> SubExceptionB - specific problem B >> >> It works for checked and unchecked exceptions, for methods in classes and interfaces. >> >> >> If it's the first time you hear about that behavior and this change affects your area, it's a good opportunity to reflect on whether the exception documentation this change adds is really needed or you were simply unaware of the fact that it was automatically added before. If exception documentation is not needed, please comment on this PR. Otherwise, consider approving it. >> >> Keep in mind that if some exception documentation is not needed, **leaving it out** from this PR might require a CSR. >> >> >> [^1]: Aside from insignificant changes on class-use and index pages. There's one relatively significant change. This change is in jdk.jshell, where some methods disappeared from "Methods declared in ..." section and other irregularities. We are aware of this and have filed JDK-8291803 to fix the root cause. >> [^2]: In reality, the feature can be suppressed, but it looks like a bug rather than intentional design. If we legitimize the feature and its suppression behavior, the model for exception documentation inheritance might become much more complicated. > > Pavel Rappo has updated the pull request incrementally with one additional commit since the last revision: > > revert an extraneous change to jdk.javadoc src/java.desktop/share/classes/javax/imageio/stream/ImageInputStreamImpl.java line 206: > 204: * @throws EOFException {@inheritDoc} > 205: */ > 206: public boolean readBoolean() throws IOException { If the docs end up the same as you say that is fine by me since for the two ImageIO classes that seems to be what we'd want. But inheritDoc behaviour is still "surprising". I've never been sure I understood it and now I'm just sure I don't. 1) The two ImageIO methods don't have @override or *anything* and yet javadoc infers they'd want to inherit the docs from the interface .. clever javadoc .. I guess I can see that some doc is better than none so this is OK 2) When it does inherit I'd expected that it copies the EXACT and ENTIRE javadoc from the super-interface, which for your change to work can't be all its doing. You've added JUST /** * @throws SomeExceptionSubType blah-blah */ and yet javadoc somehow figures out this is to be ADDED to the super-type doc for the method and not replace it .. ??? 3) What the old code was doing seems to me to be "natural" to the extent any of this does and the fix you are preparing would add to the surprising behaviours .. ------------- PR: https://git.openjdk.org/jdk/pull/10449 From djelinski at openjdk.org Fri Sep 30 20:25:27 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Fri, 30 Sep 2022 20:25:27 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v5] In-Reply-To: References: Message-ID: > Please review this test-only change that improves the execution speed of a few FileChannel tests: > - Transfer2GPlus, Transfer4GBFile, and TransferTo6GBFile were modified to use sparse files. Their speed is now consistent across platforms, provided that the platform / filesystem supports sparse files. > - LargeMapTest was rewritten to use sparse files, and to more precisely target the issue it was written to detect. In my tests it still crashed with EXCEPTION_ACCESS_VIOLATION when JDK-8286637 was reverted. > - MapTest: repetition was removed from `testForce` method; I'm not sure what issues it was supposed to catch, but at far as I can tell, it was only triggering timeouts, see [JDK-8289526](https://bugs.openjdk.org/browse/JDK-8289526), [JDK-8224480](https://bugs.openjdk.org/browse/JDK-8224480) Daniel Jeli?ski has updated the pull request incrementally with one additional commit since the last revision: Use try with resources ------------- Changes: - all: https://git.openjdk.org/jdk/pull/10464/files - new: https://git.openjdk.org/jdk/pull/10464/files/a8a28f1d..0f22fb54 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=10464&range=04 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=10464&range=03-04 Stats: 88 lines in 1 file changed: 5 ins; 11 del; 72 mod Patch: https://git.openjdk.org/jdk/pull/10464.diff Fetch: git fetch https://git.openjdk.org/jdk pull/10464/head:pull/10464 PR: https://git.openjdk.org/jdk/pull/10464 From djelinski at openjdk.org Fri Sep 30 20:25:30 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Fri, 30 Sep 2022 20:25:30 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v4] In-Reply-To: <6L0hiaFEkXLvtkxvx7dIGxO9KiDaUW3PuRxvepG02tw=.dbda303c-69f3-4a8f-9548-caadfba2b49f@github.com> References: <6L0hiaFEkXLvtkxvx7dIGxO9KiDaUW3PuRxvepG02tw=.dbda303c-69f3-4a8f-9548-caadfba2b49f@github.com> Message-ID: On Fri, 30 Sep 2022 11:07:39 GMT, Alan Bateman wrote: >> Daniel Jeli?ski has updated the pull request incrementally with one additional commit since the last revision: >> >> Fix file attributes > > test/jdk/java/nio/channels/FileChannel/TransferTo6GBFile.java line 63: > >> 61: out.println(" Writing large file..."); >> 62: long t0 = System.nanoTime(); >> 63: FileChannel fc = FileChannel.open(file, READ, WRITE); > > You can probably change this to use try-with-resources too and maybe "READ" can be dropped as this loo just writes to the file. changed to try-with-resources; READ can't be dropped - there's a `transferTo` on this channel. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From djelinski at openjdk.org Fri Sep 30 21:20:27 2022 From: djelinski at openjdk.org (Daniel =?UTF-8?B?SmVsacWEc2tp?=) Date: Fri, 30 Sep 2022 21:20:27 GMT Subject: RFR: 8294437: java/nio/channels/FileChannel tests slow on Windows [v2] In-Reply-To: References: <_YWSql2kQt0PnvC66emh7h9qnuJ-HMh1gfJ8bU-7e8o=.815afc41-c0d6-44a8-ab2d-043fb95e623e@github.com> <-5rjpuhDz69l4H16AXbDNLic9jo9qmZgD8O7oJSdT_s=.8c48d614-a6a3-4fd8-b0aa-a0040b2d07ef@github.com> Message-ID: On Fri, 30 Sep 2022 15:01:11 GMT, Brian Burkhalter wrote: >> preexisting, and unlikely to change because of Hyrum's law. I can file a ticket to update the docs if you think it makes sense. > > No, I think the docs should be left as they are. Ack that. If that's okay with you, I'd rather not change this code here; it was always like that, and I haven't found any related failure reports. The tests continue to pass. ------------- PR: https://git.openjdk.org/jdk/pull/10464 From smarks at openjdk.org Fri Sep 30 22:01:34 2022 From: smarks at openjdk.org (Stuart Marks) Date: Fri, 30 Sep 2022 22:01:34 GMT Subject: RFR: 8057113: (fs) Path should have a method to obtain the filename extension [v18] In-Reply-To: References: Message-ID: On Mon, 1 Aug 2022 21:54:22 GMT, Brian Burkhalter wrote: >> Resurrection of the proposal to add a method to obtain the filename extension originated in PR [2319](https://github.com/openjdk/jdk/pull/2319). > > Brian Burkhalter has updated the pull request incrementally with one additional commit since the last revision: > > 8057113: Remove API note OK. I think the meta-requirements are that the behavior be well-specified, implemented correctly, have tests for all the edge cases, and ideally also have defensible behavior. (I was going to write "intuitive" behavior but that's an invitation for bikeshedding, which this change has already suffered from too much.) I don't particularly care _which_ rule for special handling of `.` is used, as long as it meets the meta-requirements. I had proposed the "ignore all leading dots" rule because I thought it was simple, but I'm not beholden to this approach. If you (@bplb) prefer something different, then please go ahead with it. ------------- PR: https://git.openjdk.org/jdk/pull/8066