From szalay.beko.mate at gmail.com Thu Aug 1 09:14:39 2019 From: szalay.beko.mate at gmail.com (=?UTF-8?B?U3phbGF5LUJla8WRIE3DoXTDqQ==?=) Date: Thu, 1 Aug 2019 11:14:39 +0200 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: References: <9da8d44a-39ac-1dae-cc5d-adf5cb40e77c@oracle.com> <8C102953-1E4F-46D5-B703-4580516CA9F2@oracle.com> Message-ID: Thanks for the clarification! Indeed it should be the case. Our test behaves similarly than the minimal case sent by Chris (I also tested it with the JDK versions we use in Zookeeper). Regards, Mate On Thu, Aug 1, 2019 at 1:13 AM Alan Bateman wrote: > On 31/07/2019 15:58, Chris Hegarty wrote: > > : > > > > > > The change in behaviour comes from the `close` on the server-side > > shutting down the output before closing the socket. When debugging this > > issue the linger value being returned ( on my mac ) is -1, meaning that > > the option is disabled. > > > > I'll need to do some testing on the above patch, but on the face of it > > I believe that `shutdown` should only be called if the linger value is > > positive. > > > shutdown should only be called here if linger is disabled. I suspect the > change in behavior is that it is only done consistently on Windows in > the old SocketImpl, it is done on all platforms in the new impl. > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eolivelli at gmail.com Thu Aug 1 09:47:33 2019 From: eolivelli at gmail.com (Enrico Olivelli) Date: Thu, 1 Aug 2019 11:47:33 +0200 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: References: <9da8d44a-39ac-1dae-cc5d-adf5cb40e77c@oracle.com> <8C102953-1E4F-46D5-B703-4580516CA9F2@oracle.com> Message-ID: Il giorno gio 1 ago 2019 alle ore 11:14 Szalay-Bek? M?t? < szalay.beko.mate at gmail.com> ha scritto: > Thanks for the clarification! > > Indeed it should be the case. Our test behaves similarly than the minimal > case sent by Chris (I also tested it with the JDK versions we use in > Zookeeper). > > Regards, > Mate > > On Thu, Aug 1, 2019 at 1:13 AM Alan Bateman > wrote: > >> On 31/07/2019 15:58, Chris Hegarty wrote: >> > : >> > >> > >> > The change in behaviour comes from the `close` on the server-side >> > shutting down the output before closing the socket. When debugging this >> > issue the linger value being returned ( on my mac ) is -1, meaning that >> > the option is disabled. >> > >> > I'll need to do some testing on the above patch, but on the face of it >> > I believe that `shutdown` should only be called if the linger value is >> > positive. >> > >> shutdown should only be called here if linger is disabled. I suspect the >> change in behavior is that it is only done consistently on Windows in >> the old SocketImpl, it is done on all platforms in the new impl. >> > Just to be clear. What is the plan now ? to leave JDK13 as it is currently or to change the behavior ? If I understand correctly Alan's answer means that the new behavior in JDK13 is now consistent across Windows, Linux and Mac and we want to keep it so it is better to adapt application code. Enrico > >> -Alan >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shade at redhat.com Thu Aug 1 11:16:05 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Thu, 1 Aug 2019 13:16:05 +0200 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> Message-ID: <3a3f9422-f81d-8ee2-5c07-687f2abe18df@redhat.com> On 7/31/19 12:55 PM, Andrew Dinn wrote: >> So if pre wbsync is no-op, why do we need to handle it everywhere? We seem to be falling through all >> the way to the stub to do nothing there, maybe we should instead cut much earlier, e.g. when >> inlining Unsafe.writeBackPresync0? Would it be better to not emit CacheWBPreSync at all? > > The pre sync is definitely not needed at present. However, I put it > there because I didn't know for sure if some future port of this > capability (e.g. to ppc) might need to sync prior writes before writing > back cache lines. [Indeed, the old Intel documentation stated that > pre-sync was needed on x86 for clflush to be safe but it is definitely > not needed.] I am more concerned that the writeback call enters the pre sync stub unnecessarily. I had the idea to do this more efficiently, and simplify code at the same time: how about emitting CacheWBPreSync nodes, but emitting nothing for them in .ad matchers? That would leave generic code generic, and architectures would then be able to avoid the stub altogether for pre sync code. This would simplify current stub generators too, I think: you don't need to pass arguments to them. This leaves calling via Unsafe. I believe pulling up the isPre choice to the stub generation time would be beneficial. That is, generate *two* stubs: StubRoutines::data_cache_writeback_pre_sync() and StubRoutines::data_cache_writeback_post_sync(). If arch does not need the pre_sync, generate nop version of pre_sync(). This is not a strong requirement from my side. I do believe it would make code a bit more straight-forward. >> === src/hotspot/cpu/x86/assembler_x86.cpp >> >> It feels like these comments are redundant, especially L8630 and L8646 which mention magic values >> "6" and "7", not present in the code: ... > 8624 // 0x66 is instruction prefix > > 8627 // 0x0f 0xAE is opcode family > > 8630 // rdi == 7 is extended opcode byte > . . . > > Given that the code is simply stuffing numbers (whether supplied as > literals or as symbolic constants) into a byte stream I think these > comments are a help when it comes to cross-checking each specific > assembly against the corresponding numbers declared in the Intel > manuals. So, I don't really want to remove them. Would you prefer me to > reverse the wording as above? I was merely commenting on the style: the rest of the file does not have comments like that. The positions of prefixes, opcode families, etc is kinda implied by the code shape. >> === src/hotspot/cpu/x86/macroAssembler_x86.cpp > // prefer clwb (potentially parallel writeback without evict) > // otherwise prefer clflushopt (potentially parallel writeback > // with evict) > // otherwise fallback on clflush (serial writeback with evict) > > In the second case the comment is redundant because the need for an > sfence is covered by the existing comment inside the if: > > // need an sfence for post flush when using clflushopt or clwb > // otherwise no no need for any synchroniaztion Yes, this would be good to add. >> === src/hotspot/cpu/x86/stubGenerator_x86_64.cpp >> >> Is it really "cmpl" here, not "cmpb"? I think aarch64 code tests for byte. >> >> 2942 __ cmpl(is_pre, 0); > > This is a Java boolean input. I believe that means the value will be > loaded into c_arg0 as an int so this test ought to be adequate. Okay. >> === src/hotspot/share/opto/c2compiler.cpp >> >> Why inject new cases here, instead of at the end of switch? Saves sudden "break": >> >> 578 break; >> 579 case vmIntrinsics::_writeback0: >> 580 if (!Matcher::match_rule_supported(Op_CacheWB)) return false; >> 581 break; >> 582 case vmIntrinsics::_writebackPreSync0: >> 583 if (!Matcher::match_rule_supported(Op_CacheWBPreSync)) return false; >> 584 break; >> 585 case vmIntrinsics::_writebackPostSync0: >> 586 if (!Matcher::match_rule_supported(Op_CacheWBPostSync)) return false; >> 587 break; > > I placed them here so they were close to the other Unsafe intrinsics. In > particular they precede _allocateInstance, an ordering which is also the > case in the declarations in vmSymbols.hpp. > > In what sense do you mean that an extra 'break' is saved? That would be > true as regards the textual layout. It wouldn't affect the logic of > folding different ranges of values into branching range tests (which is > only determined by the numeric values of the intrinsics). If you are > concerned about the former then I would argue that placing the values in > declaration order seems to me to be the more important concern. I don't think we have to follow whatever ordering mess in vmSymbols.hpp. New code cuts into the last case block in that switch, which is mostly about "we know about these symbols, they are falling-through to the break". Adding cases with Matcher::match_rule_supported seems odd there. If anything, those new cases should be moved upwards to other cases, e.g. after vmIntrinsics::_minD. >> === src/hotspot/share/prims/unsafe.cpp >> >> Do we really need this function pointer mess? >> >> 457 void (*wb)(void *); >> 458 void *a = addr_from_java(line); >> 459 wb = (void (*)(void *)) StubRoutines::data_cache_writeback(); >> 460 assert(wb != NULL, "generate writeback stub!"); >> 461 (*wb)(a); >> >> Seems easier to: >> >> assert(StubRoutines::data_cache_writeback() != NULL, "sanity"); >> StubRoutines::data_cache_writeback()(addr_from_java(line)); > Hmm, "that whole brevity thing" again? Well, I guess you must now call > me "El Duderino". Well, that is, like, your opinion, man. C++ is messy if we allow it to be! > Are you sure that all the compilers used to build openJDK will happily > eat the second line of your replacement? If you can guarantee that I'll > happily remove the type declarations. I think they do: there are uses like that in the same file already, for example: if (StubRoutines::unsafe_arraycopy() != NULL) { StubRoutines::UnsafeArrayCopy_stub()(src, dst, sz); } else { Copy::conjoint_memory_atomic(src, dst, sz); } -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From adinn at redhat.com Thu Aug 1 11:48:37 2019 From: adinn at redhat.com (Andrew Dinn) Date: Thu, 1 Aug 2019 12:48:37 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> Message-ID: <04ac9cb2-4d7b-d316-9031-b4ce68c3b58e@redhat.com> Hi Boris, On 31/07/2019 13:01, Boris Ulasevich wrote: > I did a quick check of the change across our platforms. Arm32 and x86_64 > built successfully. But I see it fails due to minor issues on aarch64 > and x86_32 with webrev.09. > Can you please have a look at this? > >> src/hotspot/cpu/aarch64/aarch64.ad:2202:1: error: expected ?;? before > ?}? token >> src/hotspot/cpu/x86/macroAssembler_x86.cpp:9925: undefined reference > to `Assembler::clflush(Address)' The AArch64 error was simply a missing semi-colon. With that corrected AArch64 now builds and runs as expected (i.e. it fails the PMem support test with an UnsupportedOperationException). The second error is happening because the calling method MacroAssembler::cache_wb has not been guarded with #ifdef _LP64 (the same applies for MacroAssembler::cache_wbsync). Note that cache line writeback via Unsafe.writeBackMemory is only expected to work on Intel x86_64 so these two methods only get called from x86_64-specific code (x86_64.ad and stuGenerator_x86_64.cpp). So, the solution to this specific problem is to add #ifdef _LP64 around the declaration and implementation of these two methods. At the same time it would be helpful to remove the redundant #ifdef _LP64/#endif that I for some strange reason inserted around the definitions, but not the declarations, of clflushopt and clwb (that didn't help when I was trying to work out what was going wrong). However, a related problem also needs fixing. The Java code for method Unsafe.writebackMemory only proceeds when the data cache line writeback unit size (value of field UnsafeConstants.DATA_CACHE_LINE_FLUSH_SIZE) is non-zero. Otherwise it throws an exception. On x86_32 that field /must/ be zero. The native methods which Unsafe calls out to and the intrinsics which replace the native calls are not implemented on x86_32. The field from which the value of the Java constant is derived is currently initialised using CPU-specific information in vm_version_x86.cpp as follows if (os::supports_map_sync()) { // publish data cache line flush size to generic field, otherwise // let if default to zero thereby disabling writeback _data_cache_line_flush_size = _cpuid_info.std_cpuid1_ebx.bits.clflush_size * 8; } i.e. writeback is enabled on x86 when the operating is known to be capable of supporting MAP_SYNC. os_linux.cpp returns true for that call, irrespective of whether this is 32 or 64 bit linux. The rationale is that any Linux is capable of supporting map_sync (by contrast Windows, Solaris, AIX etc currently return false). So, the above assignment also needs to be guarded by #ifdef _LP64 in order to ensure that writeback is never attempted on x86_32. Thank you for spotting these errors. I will add the relevant fixes to the next patch and add you as a reviewer. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From Alan.Bateman at oracle.com Thu Aug 1 14:01:24 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 1 Aug 2019 07:01:24 -0700 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: References: <9da8d44a-39ac-1dae-cc5d-adf5cb40e77c@oracle.com> <8C102953-1E4F-46D5-B703-4580516CA9F2@oracle.com> Message-ID: <65b27b8f-8211-6565-31da-e5e963a1a3ff@oracle.com> On 01/08/2019 02:47, Enrico Olivelli wrote: > : > > Just to be clear. What is the plan now ? to leave JDK13 as it is > currently or to change the behavior ? > If I understand correctly Alan's answer means that the new behavior in > JDK13 is now consistent across Windows, Linux and Mac and we want to > keep it > so it is better to adapt application code. The test that Chris created to demonstrate zookeeper scenario is great as it makes it clear that the issue is closing the socket with bytes in the receive queue [1]. The JDK's behavior in this scenario has always been platform specific. On Windows, with java.net.Socket, the JDK has always shutdown the connection (when linger is not enabled). On other platforms it didn't do the shutdown. There are several subtle differences in the Windows TCP implemetation that means we can never change this without breaking many deployments (and dozens of JDK tests too). To add more complexity, NIO SocketChannel has to allow for closing when the channel is registered with a Selector and that will determine whether the connection is shutdown before the final close. So I think your updated zookeeper test is correct as it caters for the different possible behaviors that have always been possible. I don't know yet if we are going to attempt to do anything for JDK 13. JDK 13 is in rampdown phase two and there is an workaround via a system property to bring back the old behavior. If we do fix it then I think the only feasible solution is to limit the shutdown to Windows as before. So we'll reply back soon on this. -Alan [1] http://cs.baylor.edu/~donahoo/practical/CSockets/TCPRST.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.x.mcmahon at oracle.com Fri Aug 2 16:39:10 2019 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Fri, 02 Aug 2019 17:39:10 +0100 Subject: RFR [13] 8229018: Switching to an infinite socket timeout on Windows leads to high CPU load Message-ID: <5D44672E.8000007@oracle.com> Could we get the following change (from Alan Bateman) reviewed please? http://cr.openjdk.java.net/~michaelm/8229018/webrev.1/index.html It is a long standing issue but was only uncovered after switching the classic networking socket impl to use NIO in JDK 13. The bug is seen when setting a finite timeout on a socket and then switching back to block forever, which results in a busy loop. There is no regression test as it is difficult to test this kind of issue which only results in CPU overhead. Thanks, Michael From Alan.Bateman at oracle.com Sun Aug 4 19:07:07 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 4 Aug 2019 20:07:07 +0100 Subject: RFR [13] 8229018: Switching to an infinite socket timeout on Windows leads to high CPU load In-Reply-To: <5D44672E.8000007@oracle.com> References: <5D44672E.8000007@oracle.com> Message-ID: <18b02ee4-eeab-759c-c83c-e0acf519e800@oracle.com> On 02/08/2019 09:39, Michael McMahon wrote: > Could we get the following change (from Alan Bateman) reviewed please? > > http://cr.openjdk.java.net/~michaelm/8229018/webrev.1/index.html > > It is a long standing issue but was only uncovered after switching the > classic > networking socket impl to use NIO in JDK 13. The bug is seen when > setting a finite timeout > on a socket and then switching back to block forever, which results in > a busy loop. > > There is no regression test as it is difficult to test this kind of > issue which > only results in CPU overhead. Thanks for the taking this one. The Net.poll method wasn't previously used to block indefinitely so this wasn't noticed before. As you say, this is impossible to test as it isn't a correctness issue. Ditto for the untimed accept after a timed accept case where the same busy loop will happen. -Alan From christoph.langer at sap.com Mon Aug 5 08:23:16 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Mon, 5 Aug 2019 08:23:16 +0000 Subject: RFR 8213031: (zipfs) Add support for POSIX file permissions In-Reply-To: References: <953449f0913340f6a94ae3d83611fd92@sap.com> <9b3c9cfe-63e9-94ea-1af0-7ba9e2db52fd@oracle.com> <62a26037-a991-dc31-a972-a82386f63b92@oracle.com> <9806f4b1-9b55-d1d8-4511-5db0ef6786a5@oracle.com> <33E52896-E491-4660-A932-46D5A270A63E@oracle.com> <458d4ce8-d669-3117-908d-687dae2ad8d2@oracle.com> Message-ID: Hi, the changeset for the zipfs POSIX permissions support seems to be ready now. I've spent some iterations with Lance to finalize it, mainly some formal things and additions/cleanups to the test. I've got reviews now from both, Alan and Lance. So, that's the last call for review/feedback/objections from anybody else. If I don't get further feedback I'm going to push this within the next 2-3 days: http://cr.openjdk.java.net/~clanger/webrevs/8213031.17/ Thank you, Christoph From: Langer, Christoph Sent: Mittwoch, 5. Juni 2019 16:45 To: Alan Bateman ; Lance Andersen Cc: nio-dev ; Java Core Libs Subject: RE: RFR 8213031: (zipfs) Add support for POSIX file permissions Hi Lance, Alan, et al., as it seems we are getting close to CSR approval, the changeset should also be finally reviewed. Here is the most current version: http://cr.openjdk.java.net/~clanger/webrevs/8213031.14/ I have made some updates to ZipConstants as suggested by Lance in a private mail. Best regards Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From adinn at redhat.com Tue Aug 6 12:12:48 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 6 Aug 2019 13:12:48 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <3a3f9422-f81d-8ee2-5c07-687f2abe18df@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> <3a3f9422-f81d-8ee2-5c07-687f2abe18df@redhat.com> Message-ID: Hi Aleksey/Boris, This is a response to both your last review posts. New webrev link is at the end. On 01/08/2019 12:16, Aleksey Shipilev wrote: > On 7/31/19 12:55 PM, Andrew Dinn wrote: . . . > I am more concerned that the writeback call enters the pre sync stub unnecessarily. The stub? I hope you mean when executing the native call as opposed to the JITted intrinsic? The stub is only called on a cold path when a native call proper happens. By contrast, the intrinsic translation for CacheWBPreSync on AArch64 and x86_64 does not insert any instructions into the generated code (and most especially not a call to the stub). Here it is: instruct cacheWBPreSync() %{ predicate(VM_Version::supports_data_cache_line_flush()); match(CacheWBPreSync); ins_cost(100); format %{"cache wb presync" %} ins_encode %{ __ cache_wbsync(true); %} ins_pipe(pipe_slow); // XXX %} void MacroAssembler::cache_wbsync(bool is_pre) { assert(VM_Version::supports_clflush(), "clflush should be available"); bool optimized = VM_Version::supports_clflushopt(); bool no_evict = VM_Version::supports_clwb(); // pick the correct implementation if (!is_pre && (optimized || no_evict)) { // need an sfence for post flush when using clflushopt or clwb // otherwise no no need for any synchroniaztion sfence(); } } > I had the idea to do this more efficiently, and simplify code at the same time: how about emitting > CacheWBPreSync nodes, but emitting nothing for them in .ad matchers? That would leave generic code > generic, and architectures would then be able to avoid the stub altogether for pre sync code. This > would simplify current stub generators too, I think: you don't need to pass arguments to them. I believe the intrinsic behaviour you are asking for is effectively what is implemented (as shown above). The .ad match rules for the PreSync and PostSync nodes both call MacroAssembler::cache_wbsync. For pre-sync it emits nothing. For post-sync it emits sfence when the writeback is implemented using clwb or clflushopt and nothing if writeback relies on clflush. > This leaves calling via Unsafe. I believe pulling up the isPre choice to the stub generation time > would be beneficial. That is, generate *two* stubs: StubRoutines::data_cache_writeback_pre_sync() > and StubRoutines::data_cache_writeback_post_sync(). If arch does not need the pre_sync, generate nop > version of pre_sync(). I don't really see any point to doing this. We can generate two stubs, one executing a nop and one executing a nop/sfence according to need. Or we can have one stub with a branch on the sync type and branch targets that execute either a nop or a nop/sfence as needed. The difference in performance of the stub is minor and irrelevant. The difference in generation time and memory use is minor and irrelevant. What are you trying to gain here? > This is not a strong requirement from my side. I do believe it would make code a bit more > straight-forward. Am I missing something here? Or did you simply miss that the intrinsic translation inserts no code for the presync? >>> === src/hotspot/cpu/x86/assembler_x86.cpp >>> >>> It feels like these comments are redundant, especially L8630 and L8646 which mention magic values >>> "6" and "7", not present in the code: > > ... > >> 8624 // 0x66 is instruction prefix >> >> 8627 // 0x0f 0xAE is opcode family >> >> 8630 // rdi == 7 is extended opcode byte >> . . . >> >> Given that the code is simply stuffing numbers (whether supplied as >> literals or as symbolic constants) into a byte stream I think these >> comments are a help when it comes to cross-checking each specific >> assembly against the corresponding numbers declared in the Intel >> manuals. So, I don't really want to remove them. Would you prefer me to >> reverse the wording as above? > > I was merely commenting on the style: the rest of the file does not have comments like that. The > positions of prefixes, opcode families, etc is kinda implied by the code shape. Yes, I too noticed that the rest of the file does not have any such comments :-] Given the highly variable shape of x86 machine code, I don't see any reason not to start remedying that omission, even if the remedy is only piecemeal. Commenting may not be a great help to maintainers who know the code and ISA really well but they are not the only audience. Even in that specific case the comments provide a sanity check. >>> === src/hotspot/cpu/x86/macroAssembler_x86.cpp >> // prefer clwb (potentially parallel writeback without evict) >> // otherwise prefer clflushopt (potentially parallel writeback >> // with evict) >> // otherwise fallback on clflush (serial writeback with evict) >> >> In the second case the comment is redundant because the need for an >> sfence is covered by the existing comment inside the if: >> >> // need an sfence for post flush when using clflushopt or clwb >> // otherwise no no need for any synchroniaztion > > Yes, this would be good to add. Ok, done. >>> === src/hotspot/share/opto/c2compiler.cpp >>> >>> Why inject new cases here, instead of at the end of switch? Saves sudden "break": >>> >>> 578 break; >>> 579 case vmIntrinsics::_writeback0: >>> 580 if (!Matcher::match_rule_supported(Op_CacheWB)) return false; >>> 581 break; >>> 582 case vmIntrinsics::_writebackPreSync0: >>> 583 if (!Matcher::match_rule_supported(Op_CacheWBPreSync)) return false; >>> 584 break; >>> 585 case vmIntrinsics::_writebackPostSync0: >>> 586 if (!Matcher::match_rule_supported(Op_CacheWBPostSync)) return false; >>> 587 break; >> >> I placed them here so they were close to the other Unsafe intrinsics. In >> particular they precede _allocateInstance, an ordering which is also the >> case in the declarations in vmSymbols.hpp. >> >> In what sense do you mean that an extra 'break' is saved? That would be >> true as regards the textual layout. It wouldn't affect the logic of >> folding different ranges of values into branching range tests (which is >> only determined by the numeric values of the intrinsics). If you are >> concerned about the former then I would argue that placing the values in >> declaration order seems to me to be the more important concern. > > I don't think we have to follow whatever ordering mess in vmSymbols.hpp. New code cuts into the last > case block in that switch, which is mostly about "we know about these symbols, they are > falling-through to the break". Adding cases with Matcher::match_rule_supported seems odd there. If > anything, those new cases should be moved upwards to other cases, e.g. after vmIntrinsics::_minD. As you wish. I have moved them to immediately preceding the large unbroken block. >>> === src/hotspot/share/prims/unsafe.cpp >>> >>> Do we really need this function pointer mess? >>> >>> 457 void (*wb)(void *); >>> 458 void *a = addr_from_java(line); >>> 459 wb = (void (*)(void *)) StubRoutines::data_cache_writeback(); >>> 460 assert(wb != NULL, "generate writeback stub!"); >>> 461 (*wb)(a); >>> >>> Seems easier to: >>> >>> assert(StubRoutines::data_cache_writeback() != NULL, "sanity"); >>> StubRoutines::data_cache_writeback()(addr_from_java(line)); >> Hmm, "that whole brevity thing" again? Well, I guess you must now call >> me "El Duderino". > > Well, that is, like, your opinion, man. C++ is messy if we allow it to be! > >> Are you sure that all the compilers used to build openJDK will happily >> eat the second line of your replacement? If you can guarantee that I'll >> happily remove the type declarations. > > I think they do: there are uses like that in the same file already, for example: > > if (StubRoutines::unsafe_arraycopy() != NULL) { > StubRoutines::UnsafeArrayCopy_stub()(src, dst, sz); > } else { > Copy::conjoint_memory_atomic(src, dst, sz); > } Hmm, your suggested replacement does not in fact compile. Indeed, the example you cite is comparing apples with pears -- note that the getter employed in call differs from the getter employed in the check. The following macro magic is provided for that example in stubRoutines.hpp: static address unsafe_arraycopy() { return _unsafe_arraycopy; } typedef void (*UnsafeArrayCopyStub)(const void* src, void* dst, size_t count); static UnsafeArrayCopyStub UnsafeArrayCopy_stub() { return CAST_TO_FN_PTR(UnsafeArrayCopyStub, _unsafe_arraycopy); } I have provided similar magic to hide the function pointer details for the writeback and writeback_sync stubs. Latest webrev against current jdk/jdk including changes agreed for the review discussion above fixes for AArch64/x86_32 build issues reported by Boris Ulasevich the new test promised for Boris Ulasevich http://cr.openjdk.java.net/~adinn/8224974/webrev.11 Testing: The x86_32 and aarch64 builds now build and run ok. The pmem-specific tests (PmemTest, MapFail) pass with the expected outcomes on x86_64 and AArch64. PmemTest is skipped on x86_32 (as expected). MapFail passes on x86_32 (it expects the map to be unsupported). I have passed the patch on for more thorough testing on simulated and real NVRAM using our middleware stack. I am still waiting for confirmation of a submit job. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From shade at redhat.com Tue Aug 6 12:44:19 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 6 Aug 2019 14:44:19 +0200 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> <3a3f9422-f81d-8ee2-5c07-687f2abe18df@redhat.com> Message-ID: On 8/6/19 2:12 PM, Andrew Dinn wrote: > On 01/08/2019 12:16, Aleksey Shipilev wrote: >> On 7/31/19 12:55 PM, Andrew Dinn wrote: >> I am more concerned that the writeback call enters the pre sync stub unnecessarily. > > The stub? I hope you mean when executing the native call as opposed to > the JITted intrinsic? The stub is only called on a cold path when a > native call proper happens. By contrast, the intrinsic translation for > CacheWBPreSync on AArch64 and x86_64 does not insert any instructions > into the generated code (and most especially not a call to the stub). Ah, that is exactly what I wanted. Good then, scratch the rest of my comments. >> This is not a strong requirement from my side. I do believe it would make code a bit more >> straight-forward. > > Am I missing something here? Or did you simply miss that the intrinsic > translation inserts no code for the presync? I thought that translating two separate (and statically bound) Unsafe calls, hooking them up to separate Unsafe leaf entries, and then suddenly going into a single StubRoutine call with dynamic argument that dispatches at runtime is a bit awkward. I would have expected it to end up with two separate StubRoutines as well. Again, I have no strong opinion about this. >>>> === src/hotspot/share/prims/unsafe.cpp >>>> >>>> Do we really need this function pointer mess? >>>> >>>> 457 void (*wb)(void *); >>>> 458 void *a = addr_from_java(line); >>>> 459 wb = (void (*)(void *)) StubRoutines::data_cache_writeback(); >>>> 460 assert(wb != NULL, "generate writeback stub!"); >>>> 461 (*wb)(a); >>>> >>>> Seems easier to: ... > static address unsafe_arraycopy() { return _unsafe_arraycopy; } > typedef void (*UnsafeArrayCopyStub)(const void* src, void* dst, size_t > count); > static UnsafeArrayCopyStub UnsafeArrayCopy_stub() { return > CAST_TO_FN_PTR(UnsafeArrayCopyStub, _unsafe_arraycopy); } > > I have provided similar magic to hide the function pointer details for > the writeback and writeback_sync stubs. Yes, this looks cleaner. The declarations can be a bit less crowded: static address data_cache_writeback() { return _data_cache_writeback; } static address data_cache_writeback_sync() { return _data_cache_writeback_sync; } typedef void (*DataCacheWritebackStub)(void *); static DataCacheWritebackStub DataCacheWriteback_stub() { return ... typedef void (*DataCacheWritebackSyncStub)(bool); static DataCacheWritebackSyncStub DataCacheWritebackSync_stub() { return ... > http://cr.openjdk.java.net/~adinn/8224974/webrev.11 Looks good. Minor nits (no need for another webrev): *) Not sure if the only copyright line change is needed in src/hotspot/cpu/aarch64/globals_aarch64.hpp. *) Indenting is a bit off at L109 in src/hotspot/cpu/aarch64/vm_version_aarch64.hpp: 108 static int cpu_revision() { return _revision; } 109 static bool supports_dcpop() { return _dcpop; } *) Excess new line added at the end of src/hotspot/os/bsd/os_bsd.cpp? *) Indenting is off in backslashes in src/hotspot/share/runtime/globals.hpp: 2444 \ 2445 develop(bool, TraceMemoryWriteback, false, \ 2446 "Trace memory writeback operations") \ 2447 \ *) Unnecessary newline at L827 in src/hotspot/share/runtime/os.hpp? 826 // support for mapping non-volatile memory using MAP_SYNC 827 828 static bool supports_map_sync(); *) These declarations are too dense in src/java.base/share/classes/jdk/internal/misc/Unsafe.java: 998 /** 999 * primitive operation forcing writeback of a single cache line. 1000 * 1001 * @param address 1002 * the start address of the cache line to be written back 1003 */ 1004 // native used to write back an individual cache line starting at 1005 // the supplied address 1006 @HotSpotIntrinsicCandidate 1007 private native void writeback0(long address); 1008 // native used to serialise writeback operations relative to 1009 // preceding memory writes 1010 @HotSpotIntrinsicCandidate 1011 private native void writebackPreSync0(); 1012 // native used to serialise writeback operations relative to 1013 // following memory writes 1014 @HotSpotIntrinsicCandidate 1015 private native void writebackPostSync0(); Suggestion: /** * Force write back an individual cache line. * * @param address * the start address of the cache line to be written back */ @HotSpotIntrinsicCandidate private native void writeback0(long address); /** * Serialize writeback operations relative to preceding memory writes. */ @HotSpotIntrinsicCandidate private native void writebackPreSync0(); /** * Serialize writeback operations relative to following memory writes. */ @HotSpotIntrinsicCandidate private native void writebackPostSync0(); -- Thanks, -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From adinn at redhat.com Tue Aug 6 13:57:05 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 6 Aug 2019 14:57:05 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> <3a3f9422-f81d-8ee2-5c07-687f2abe18df@redhat.com> Message-ID: <67c3888c-9058-9e5c-273e-7810b1c66e29@redhat.com> On 06/08/2019 13:44, Aleksey Shipilev wrote: > Ah, that is exactly what I wanted. Good then, scratch the rest of my comments. > . . . > I thought that translating two separate (and statically bound) Unsafe calls, hooking them up to > separate Unsafe leaf entries, and then suddenly going into a single StubRoutine call with dynamic > argument that dispatches at runtime is a bit awkward. I would have expected it to end up with two > separate StubRoutines as well. Again, I have no strong opinion about this. Ok, thanks for clarifying. Inertia dictates I leave the stubs as is :-) > Yes, this looks cleaner. The declarations can be a bit less crowded: > > static address data_cache_writeback() { return _data_cache_writeback; } > static address data_cache_writeback_sync() { return _data_cache_writeback_sync; } > > typedef void (*DataCacheWritebackStub)(void *); > static DataCacheWritebackStub DataCacheWriteback_stub() { return ... > > typedef void (*DataCacheWritebackSyncStub)(bool); > static DataCacheWritebackSyncStub DataCacheWritebackSync_stub() { return ... > . . . >> http://cr.openjdk.java.net/~adinn/8224974/webrev.11 > > Looks good. Ok, I'll fold this and the other format errors you identified into the next patch. If I could please get a nod from Alan Bateman (and assuming I don't receive further comments from other reviewers) I'll push that next patch. Any more for any more ... ? regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From adinn at redhat.com Tue Aug 6 16:09:46 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 6 Aug 2019 17:09:46 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> <3a3f9422-f81d-8ee2-5c07-687f2abe18df@redhat.com> <67c3888c-9058-9e5c-273e-7810b1c66e29@redhat.com> Message-ID: <3fb6e11f-4c95-8d88-3144-0c46c455a5db@redhat.com> Hello Dmitry, On 06/08/2019 15:25, Dmitry Chuyko wrote: > One quick question about synchronization in unmappers. One of > preliminary steps for Loom was to replace monitor usage by j.u.c locks > for I/O to let fibers release carrier threads. For instance see > JDK-8222774. Does it make sense to do the same in your new unmappers code? > . . . > [1] https://bugs.openjdk.java.net/browse/JDK-8222774 The unmapper code is not strictly 'new' as regards its reliance on synchronization. It merely follows and repeats the pattern employed in the prior code that it has generalized (by splitting the original Unmapper into two distinct flavours of subclass). If this poses a problem for Loom then it is a separate issue form the one this JEP addresses. I think you should raise a new issue for that change (just as you would have had to do before this change). I am sure Alan Bateman will be happy to consider your proposal. Indeed, I would be happy to implement it given his approval -- or leave it to you to do so if you prefer. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From dmitry.chuyko at bell-sw.com Wed Aug 7 09:44:51 2019 From: dmitry.chuyko at bell-sw.com (Dmitry Chuyko) Date: Wed, 7 Aug 2019 12:44:51 +0300 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <3fb6e11f-4c95-8d88-3144-0c46c455a5db@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> <3a3f9422-f81d-8ee2-5c07-687f2abe18df@redhat.com> <67c3888c-9058-9e5c-273e-7810b1c66e29@redhat.com> <3fb6e11f-4c95-8d88-3144-0c46c455a5db@redhat.com> Message-ID: <83466f7a-0436-940c-ca87-2a8b4a7c3e7d@bell-sw.com> On 8/6/19 7:09 PM, Andrew Dinn wrote: > Hello Dmitry, > > On 06/08/2019 15:25, Dmitry Chuyko wrote: >> One quick question about synchronization in unmappers. One of >> preliminary steps for Loom was to replace monitor usage by j.u.c locks >> for I/O to let fibers release carrier threads. For instance see >> JDK-8222774. Does it make sense to do the same in your new unmappers code? >> . . . >> [1] https://bugs.openjdk.java.net/browse/JDK-8222774 > The unmapper code is not strictly 'new' as regards its reliance on > synchronization. It merely follows and repeats the pattern employed in > the prior code that it has generalized (by splitting the original > Unmapper into two distinct flavours of subclass). > > If this poses a problem for Loom then it is a separate issue form the > one this JEP addresses. I think you should raise a new issue for that > change (just as you would have had to do before this change). I am sure > Alan Bateman will be happy to consider your proposal. Indeed, I would be > happy to implement it given his approval -- or leave it to you to do so > if you prefer. Agree, Loom has a long road to go. So I suppose such a change will be a part of larger work in sun.nio, and I or one of my colleagues will be happy to participate. Changes will probably be straightforward (e.g. JDK-8222882) but this synchronization is not covered by regression tests so I believe in this case you'll help to retry some of your ad-hoc testing or maybe some application tests you know about. -Dmitry > > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From Alan.Bateman at oracle.com Wed Aug 7 10:21:31 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 7 Aug 2019 11:21:31 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <3fb6e11f-4c95-8d88-3144-0c46c455a5db@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> <3a3f9422-f81d-8ee2-5c07-687f2abe18df@redhat.com> <67c3888c-9058-9e5c-273e-7810b1c66e29@redhat.com> <3fb6e11f-4c95-8d88-3144-0c46c455a5db@redhat.com> Message-ID: On 06/08/2019 09:09, Andrew Dinn wrote: > : > The unmapper code is not strictly 'new' as regards its reliance on > synchronization. It merely follows and repeats the pattern employed in > the prior code that it has generalized (by splitting the original > Unmapper into two distinct flavours of subclass). > > If this poses a problem for Loom then it is a separate issue form the > one this JEP addresses. I think you should raise a new issue for that > change (just as you would have had to do before this change). I am sure > Alan Bateman will be happy to consider your proposal. Indeed, I would be > happy to implement it given his approval -- or leave it to you to do so > if you prefer. > I don't think we need to be concerned with any of this at this time. The unmapper is run by the reference handler thread. Also the synchronization here is for the counters so not the same thing as doing a blocking I/O operation while holding a monitor. At some point we'll examine all the file I/O operations as some of these are candidates for managed blockers, others are candidates for alternative implementations - there are bigger issues to resolve first and we've been trying to avoid carrying too many changes due to the complexity and effort needed to keep them in sync with the main line. -Alan From christoph.langer at sap.com Thu Aug 8 09:32:49 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 8 Aug 2019 09:32:49 +0000 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> Message-ID: Hi Brian, after the change came in, we see sporadic test failures: java.lang.RuntimeException: Expected modification timestamp to be '1483261261123456789', but was '1483261261000000000' at SetTimesNanos.testNanos(SetTimesNanos.java:85) at SetTimesNanos.main(SetTimesNanos.java:59) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:565) at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127) at java.base/java.lang.Thread.run(Thread.java:830) JavaTest Message: Test threw exception: java.lang.RuntimeException: Expected modification timestamp to be '1483261261123456789', but was '1483261261000000000' JavaTest Message: shutting down test I opened https://bugs.openjdk.java.net/browse/JDK-8229280 to track this. Can you please have a look? Thanks Christoph From: nio-dev On Behalf Of Brian Burkhalter Sent: Freitag, 19. Juli 2019 17:53 To: Alan Bateman Cc: nio-dev Subject: Re: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps On Jul 19, 2019, at 7:59 AM, Brian Burkhalter > wrote: On Jul 19, 2019, at 4:14 AM, Alan Bateman > wrote: On the test then it would be better to use the scratch directory as Lance suggests. Also would be useful to have it run a few hundred times on all platforms before publish to be confident that it is reliable (we've had a lot of issues with tests for file time stamps as you know). OK I will update the test and do a large number of test runs before pushing this. Test updated as --- a/test/jdk/java/nio/file/attribute/BasicFileAttributeView/SetTimesNanos.java +++ b/test/jdk/java/nio/file/attribute/BasicFileAttributeView/SetTimesNanos.java @@ -45,10 +45,10 @@ public static void main(String[] args) throws IOException, InterruptedException { - Path dirPath = Path.of(System.getProperty("test.dir", "."), "test"); + Path dirPath = Path.of("test"); Path dir = Files.createDirectory(dirPath); FileStore store = Files.getFileStore(dir); - System.err.format("FileStore: %s on %s (%s)%n", dir, store.name(), + System.out.format("FileStore: %s on %s (%s)%n", dir, store.name(), store.type()); if (System.getProperty("os.name").toLowerCase().startsWith("mac") && store.name().equalsIgnoreCase("hfs")) { @@ -58,7 +58,7 @@ } testNanos(dir); - Path file = Files.createTempFile(dir, "test", "test"); + Path file = Files.createFile(dir.resolve("test.dat")); testNanos(file); } Build-and-test job submitted; some test-only jobs to follow. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Aug 8 10:03:36 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 8 Aug 2019 11:03:36 +0100 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> Message-ID: <7b4c2bd4-4e3d-70f0-9d8b-b0d24ae038a0@oracle.com> On 08/08/2019 10:32, Langer, Christoph wrote: > > Hi Brian, > > after the change came in, we see sporadic test failures: > > java.lang.RuntimeException: Expected modification timestamp to be > '1483261261123456789', but was '1483261261000000000' > Which operating system is this? The actual timestamp is truncated which suggests that it doesn't support nanos. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Thu Aug 8 14:56:51 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 8 Aug 2019 14:56:51 +0000 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: <7b4c2bd4-4e3d-70f0-9d8b-b0d24ae038a0@oracle.com> References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> <7b4c2bd4-4e3d-70f0-9d8b-b0d24ae038a0@oracle.com> Message-ID: Hi Alan, it seems that we're encountering this behavior on old Suse SLES 11.3 and 11.4 machines, both ppc64 and linuxx86_64. Maybe these can somehow be excluded from the test? Or the test can detect upfront when nanotime isn't supported? /Christoph From: Alan Bateman Sent: Donnerstag, 8. August 2019 12:04 To: Langer, Christoph ; Brian Burkhalter Cc: nio-dev Subject: Re: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps On 08/08/2019 10:32, Langer, Christoph wrote: Hi Brian, after the change came in, we see sporadic test failures: java.lang.RuntimeException: Expected modification timestamp to be '1483261261123456789', but was '1483261261000000000' Which operating system is this? The actual timestamp is truncated which suggests that it doesn't support nanos. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Aug 8 15:52:33 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 8 Aug 2019 08:52:33 -0700 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> <7b4c2bd4-4e3d-70f0-9d8b-b0d24ae038a0@oracle.com> Message-ID: HI Christoph, I don?t know of a programmatic way to check whether nanosecond precision is supported. Do you happen to know the file system type (and os.name property)? Right now the test has this exclusion 53 if (System.getProperty("os.name").toLowerCase().startsWith("mac") && 54 store.name().equalsIgnoreCase("hfs")) { 55 System.err.println 56 ("HFS on macOS does not have nsec timestamps: skipping test"); 57 return; 58 } Thanks, Brian > On Aug 8, 2019, at 7:56 AM, Langer, Christoph wrote: > > it seems that we?re encountering this behavior on old Suse SLES 11.3 and 11.4 machines, both ppc64 and linuxx86_64. > > Maybe these can somehow be excluded from the test? Or the test can detect upfront when nanotime isn?t supported? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Aug 9 12:12:38 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 9 Aug 2019 13:12:38 +0100 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: References: <9da8d44a-39ac-1dae-cc5d-adf5cb40e77c@oracle.com> <8C102953-1E4F-46D5-B703-4580516CA9F2@oracle.com> Message-ID: On 01/08/2019 10:47, Enrico Olivelli wrote: > > Just to be clear. What is the plan now ? to leave JDK13 as it is > currently or to change the behavior ? > Just getting back to this one. I think we've established in this discussion here that the behavior change arises when closing a TCP socket with bytes in the receive queue. The JDK behavior in this scenario varies by platform and release. It also varies depending on whether SO_LINGER is enabled and the linger value. Additionally for SocketChannels, it depends on whether the channel is registered with a Selector or not. Here's the behavior of close with unread data with JDK 8, 11, and 13 : JDK 8 ==== java.net.Socket ? Windows: hard close if linger enabled, graceful close if linger disabled ? Linux/macOS: hard close SocketChannel ? Windows: hard close if linger enabled, graceful close if linger disabled ? Linux/macOS: hard close JDK 11 ==== java.net.Socket ? Windows: hard close if linger enabled, graceful close if linger disabled ? Linux/macOS: hard close SocketChannel ? All platforms: hard close if linger enabled with a value of 0 or not registered with a Selector, graceful close otherwise JDK 13 ==== java.net.Socket ? All platforms: hard close if linger enabled with a value of 0, graceful close otherwise SocketChannel ? All platforms: hard close if linger enabled with a value of 0 or not registered with a Selector, graceful close otherwise As you can see, there is a matrix of scenarios so having the Zookeeper test handle both a graceful close and the server resetting the connection means it should work in all scenarios. I discussed briefly the subtle behavior change in JDK 13 with Chris and I and I think we've agreed that we shouldn't attempt to change the JDK 13 behavior (JDK 13 has since got to RC phase and this isn't a showstopper). One nice thing about the JDK 13 behavior is that it is consistent across platforms so it may be that we should just leave it as it and just document this as another behavior change in the JEP. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Aug 9 16:17:45 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 9 Aug 2019 09:17:45 -0700 Subject: 8225461: (fs) Investigate filtering duplicates out of Iterable returned by java.nio.file.FileSystem::getFileStores In-Reply-To: References: <9d73d427-f83a-cedc-45cb-1c6fa1299306@oracle.com> Message-ID: <13876EA1-D51B-4B67-ADED-C7174D46235F@oracle.com> > On Jun 11, 2019, at 10:25 AM, Brian Burkhalter wrote: > >> On Jun 11, 2019, at 3:46 AM, Alan Bateman > wrote: >> >>> I don?t know whether the specification of FileSystem::getFileStores() should be updated to indicate that no duplicate stores are returned, which would necessitate a CSR. >>> >> This issue is going a bit of time to review as it changes the meaning of equals/hashCode and has several other implications that will take time to work through. I will try to get to it soon, but maybe not this week. > > Understood. Note that the new equals() and hashCode() methods in UnixMountEntry are identical to the respective methods of UnixFileStore. Reprising the thread about [1] starting from the message [2]. Un updated patch is at [3]. Contrary to the previous version, this patch does not change any implementation code and does *not* attempt to keep duplicates out of the results of getFileStores(). Instead this is allowed to remain a reflection of the contents of /proc/mounts (or /etc/mtab) on Linux. The test code is restructured to replace the FileUtils method areAllMountPointsAccessible() with getNumReplicateMountPoints(). If duplicate FileStores are detected, then no exception is thrown unless the number of replicates does not match that obtained directly from /proc/mounts. This approach should also address [4]. One thing I noticed which I don?t know is correct is that sun.nio.fs.LinuxFileSystem.getMountEntries() uses /etc/mtab but sun.nio.fs.LinuxFileStore.findMountEntry() uses /proc/mounts. On my local Linux VM both of these point to /proc/self/mounts but that might not be the case on other systems. Thanks, Brian [1] https://bugs.openjdk.java.net/browse/JDK-8225461 [2] http://mail.openjdk.java.net/pipermail/nio-dev/2019-June/006282.html [3] http://cr.openjdk.java.net/~bpb/8225461/webrev.01/ [4] https://bugs.openjdk.java.net/browse/JDK-8225471 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Aug 9 16:18:04 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 9 Aug 2019 09:18:04 -0700 Subject: 6937320: (fs) Path getFileStore().type() does not return correct type of file store on Linux Message-ID: <9A8E7D3B-9085-4F83-8AEE-B56263BE19FA@oracle.com> https://bugs.openjdk.java.net/browse/JDK-6937320 http://cr.openjdk.java.net/~bpb/6937320/webrev.00/ The failure addressed here occurred on Linux when trying to obtain the FileStore of ?/? which on some misconfigured systems was ?rootfs? instead of the actual device due to duplicate entries for ?/? in /proc/mounts with the rootfs entry being first. The change is to attempt to recognize an actual device path, e.g., /dev/sda1, in the entry and to use the first non-path entry, e.g., rootfs, only if a path-like entry was not encountered first. Thanks, Brian From ecki at zusammenkunft.net Fri Aug 9 17:35:28 2019 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 9 Aug 2019 17:35:28 +0000 Subject: 6937320: (fs) Path getFileStore().type() does not return correct type of file store on Linux In-Reply-To: <9A8E7D3B-9085-4F83-8AEE-B56263BE19FA@oracle.com> References: <9A8E7D3B-9085-4F83-8AEE-B56263BE19FA@oracle.com> Message-ID: Hello, The bug title is a bit misleading, can that be fixed before it shows up in the commit? Gruss Bernd -- http://bernd.eckenfels.net ________________________________ Von: nio-dev im Auftrag von Brian Burkhalter Gesendet: Freitag, August 9, 2019 7:18 PM An: nio-dev Betreff: 6937320: (fs) Path getFileStore().type() does not return correct type of file store on Linux https://bugs.openjdk.java.net/browse/JDK-6937320 http://cr.openjdk.java.net/~bpb/6937320/webrev.00/ The failure addressed here occurred on Linux when trying to obtain the FileStore of ?/? which on some misconfigured systems was ?rootfs? instead of the actual device due to duplicate entries for ?/? in /proc/mounts with the rootfs entry being first. The change is to attempt to recognize an actual device path, e.g., /dev/sda1, in the entry and to use the first non-path entry, e.g., rootfs, only if a path-like entry was not encountered first. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecki at zusammenkunft.net Fri Aug 9 18:49:35 2019 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 9 Aug 2019 18:49:35 +0000 Subject: 8225461: (fs) Investigate filtering duplicates out of Iterable returned by java.nio.file.FileSystem::getFileStores In-Reply-To: <13876EA1-D51B-4B67-ADED-C7174D46235F@oracle.com> References: <9d73d427-f83a-cedc-45cb-1c6fa1299306@oracle.com> , <13876EA1-D51B-4B67-ADED-C7174D46235F@oracle.com> Message-ID: Hello, I wonder if for the df case you should word-split the mountpoints, assuming that if duplicate lines are possible they might differ in any of the other columns which are not relevant for the test. (And i wonder if this as well as the /proc/mounts Support Blanks in the path) Speaking of mountpoints, it looks like FileStore.toStrimg is still the only way to get the mountpoint path (also for WindowsFilestore), would it be good to get that with an accessor method? -- http://bernd.eckenfels.net ________________________________ Von: nio-dev im Auftrag von Brian Burkhalter Gesendet: Freitag, August 9, 2019 7:48 PM An: nio-dev Betreff: Re: 8225461: (fs) Investigate filtering duplicates out of Iterable returned by java.nio.file.FileSystem::getFileStores On Jun 11, 2019, at 10:25 AM, Brian Burkhalter > wrote: On Jun 11, 2019, at 3:46 AM, Alan Bateman > wrote: I don?t know whether the specification of FileSystem::getFileStores() should be updated to indicate that no duplicate stores are returned, which would necessitate a CSR. This issue is going a bit of time to review as it changes the meaning of equals/hashCode and has several other implications that will take time to work through. I will try to get to it soon, but maybe not this week. Understood. Note that the new equals() and hashCode() methods in UnixMountEntry are identical to the respective methods of UnixFileStore. Reprising the thread about [1] starting from the message [2]. Un updated patch is at [3]. Contrary to the previous version, this patch does not change any implementation code and does *not* attempt to keep duplicates out of the results of getFileStores(). Instead this is allowed to remain a reflection of the contents of /proc/mounts (or /etc/mtab) on Linux. The test code is restructured to replace the FileUtils method areAllMountPointsAccessible() with getNumReplicateMountPoints(). If duplicate FileStores are detected, then no exception is thrown unless the number of replicates does not match that obtained directly from /proc/mounts. This approach should also address [4]. One thing I noticed which I don?t know is correct is that sun.nio.fs.LinuxFileSystem.getMountEntries() uses /etc/mtab but sun.nio.fs.LinuxFileStore.findMountEntry() uses /proc/mounts. On my local Linux VM both of these point to /proc/self/mounts but that might not be the case on other systems. Thanks, Brian [1] https://bugs.openjdk.java.net/browse/JDK-8225461 [2] http://mail.openjdk.java.net/pipermail/nio-dev/2019-June/006282.html [3] http://cr.openjdk.java.net/~bpb/8225461/webrev.01/ [4] https://bugs.openjdk.java.net/browse/JDK-8225471 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Aug 9 18:54:10 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 9 Aug 2019 11:54:10 -0700 Subject: 8225461: (fs) Investigate filtering duplicates out of Iterable returned by java.nio.file.FileSystem::getFileStores In-Reply-To: References: <9d73d427-f83a-cedc-45cb-1c6fa1299306@oracle.com> <13876EA1-D51B-4B67-ADED-C7174D46235F@oracle.com> Message-ID: <47D28DD6-B089-471E-A2D3-C13362ED8B3F@oracle.com> Hello, > On Aug 9, 2019, at 11:49 AM, Bernd Eckenfels wrote: > > I wonder if for the df case you should word-split the mountpoints, assuming that if duplicate lines are possible they might differ in any of the other columns which are not relevant for the test. (And i wonder if this as well as the /proc/mounts Support Blanks in the path) That might be a good idea. > Speaking of mountpoints, it looks like FileStore.toStrimg is still the only way to get the mountpoint path (also for WindowsFilestore), would it be good to get that with an accessor method? There is the issue [1] on file and I was wondering the same as you. Thanks, Brian [1] https://bugs.openjdk.java.net/browse/JDK-8028426 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Sat Aug 10 00:05:54 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 9 Aug 2019 17:05:54 -0700 Subject: 8229280: Test failures on several linux hosts after JDK-8181493 In-Reply-To: References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> <7b4c2bd4-4e3d-70f0-9d8b-b0d24ae038a0@oracle.com> Message-ID: <8DE2D4E9-E312-4E73-A692-DE35141BFC2A@oracle.com> Here is a potential fix [1] for the issue [2] provoked by the fix for [3]. The change is not to require strict equality in timestamps except for some file system types which are known to support nanosecond precision and are explicitly identified. For non-strict timestamp comparisons, equality is defined by the timestamp read back from the file having a number of trailing zeros which is a multiple of 3, and having a prefix which is equal to a prefix of the same length of the nanosecond precision timestamp. If there are more file system types which should be added to the list of strictly tested types, please let me know. Thanks, Brian [1] http://cr.openjdk.java.net/~bpb/8229280/webrev.00/ [2] https://bugs.openjdk.java.net/browse/JDK-8229280 [3] https://bugs.openjdk.java.net/browse/JDK-8181493 > On Aug 8, 2019, at 8:52 AM, Brian Burkhalter wrote: > > HI Christoph, > > I don?t know of a programmatic way to check whether nanosecond precision is supported. Do you happen to know the file system type (and os.name property)? Right now the test has this exclusion > 53 if (System.getProperty("os.name").toLowerCase().startsWith("mac") && > 54 store.name().equalsIgnoreCase("hfs")) { > 55 System.err.println > 56 ("HFS on macOS does not have nsec timestamps: skipping test"); > 57 return; > 58 } > Thanks, > > Brian > >> On Aug 8, 2019, at 7:56 AM, Langer, Christoph > wrote: >> >> it seems that we?re encountering this behavior on old Suse SLES 11.3 and 11.4 machines, both ppc64 and linuxx86_64. >> >> Maybe these can somehow be excluded from the test? Or the test can detect upfront when nanotime isn?t supported? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From 00jy116 at 163.com Mon Aug 12 09:49:48 2019 From: 00jy116 at 163.com (=?GBK?B?08q8/g==?=) Date: Mon, 12 Aug 2019 17:49:48 +0800 (CST) Subject: The method of Filechanely.map() would cause the JVM crash Message-ID: <2d575a3d.ca0c.16c853bdd78.Coremail.00jy116@163.com> Hi, all When using OpenJDK 1.8 8u92, we found that the method of Filechanely.map() would cause the JVM crash. And then we found a solution in :https://bugs.openjdk.java.net/browse/JDK-8168628. To verify the correctness of the solution,we had some tests besed in OpenJDK8 8u202.Unfortunately, the problem was not well solved,the JVM crash still happened(Attachment1.rar).And then, we tested it again in OpenJDK11,the JVM crash still happened too,this is the crash log:Attachment2.rar. So,the solution in: https://bugs.openjdk.java.net/browse/JDK-8168628 and related can not solve this problem thoroughly until now. Here is the test case we provided:crash-demo.rar -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Attachment1.rar Type: application/octet-stream Size: 1350 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Attachment2.rar Type: application/octet-stream Size: 10757 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: crash-demo.rar Type: application/octet-stream Size: 5193 bytes Desc: not available URL: From Alan.Bateman at oracle.com Mon Aug 12 09:59:02 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 12 Aug 2019 10:59:02 +0100 Subject: The method of Filechanely.map() would cause the JVM crash In-Reply-To: <2d575a3d.ca0c.16c853bdd78.Coremail.00jy116@163.com> References: <2d575a3d.ca0c.16c853bdd78.Coremail.00jy116@163.com> Message-ID: <0230388a-2872-eeb8-d3fe-ae70d8ec1dc9@oracle.com> On 12/08/2019 10:49, ?? wrote: > Hi, all > When using OpenJDK 1.8 8u92, we found that the method of > Filechanely.map() would cause the JVM crash. > And then we found a solution in > :https://bugs.openjdk.java.net/browse/JDK-8168628. > To verify the correctness of the solution,we had some tests besed in > OpenJDK8 8u202.Unfortunately, > the problem was not well solved,the JVM crash still > happened(Attachment1.rar).And then, > we tested it again in OpenJDK11,the JVM crash still happened too,this > is the crash log:Attachment2.rar. > So,the solution in: https://bugs.openjdk.java.net/browse/JDK-8168628 > and related can not solve this problem thoroughly until now. > Here is the test case we provided:crash-demo.rar There were several issues and side effects with JDK-8168628 so the changes had to be backed out. The underlying issue was the lack of memory guards on bulk operations and this has been fixed for JDK 14 via JDK-8191278. I realize you are testing with jdk8u builds but I think it would be useful if you could try the latest JDK 14 EA build to makes sure that the issues you are seeing no longer exist. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Aug 12 13:34:18 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 12 Aug 2019 14:34:18 +0100 Subject: 6937320: (fs) Path getFileStore().type() does not return correct type of file store on Linux In-Reply-To: <9A8E7D3B-9085-4F83-8AEE-B56263BE19FA@oracle.com> References: <9A8E7D3B-9085-4F83-8AEE-B56263BE19FA@oracle.com> Message-ID: On 09/08/2019 17:18, Brian Burkhalter wrote: > https://bugs.openjdk.java.net/browse/JDK-6937320 > http://cr.openjdk.java.net/~bpb/6937320/webrev.00/ > > The failure addressed here occurred on Linux when trying to obtain the FileStore of ?/? which on some misconfigured systems was ?rootfs? instead of the actual device due to duplicate entries for ?/? in /proc/mounts with the rootfs entry being first. The change is to attempt to recognize an actual device path, e.g., /dev/sda1, in the entry and to use the first non-path entry, e.g., rootfs, only if a path-like entry was not encountered first. > Do you have examples, or maybe any references, to how these mount entries are setup? I think we need these before deciding if changes are needed. -Alan From brian.burkhalter at oracle.com Mon Aug 12 16:01:37 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 12 Aug 2019 09:01:37 -0700 Subject: 6937320: (fs) Path getFileStore().type() does not return correct type of file store on Linux In-Reply-To: References: <9A8E7D3B-9085-4F83-8AEE-B56263BE19FA@oracle.com> Message-ID: <27070CD6-DE3F-4F4A-B24E-995B4C5BBC56@oracle.com> > On Aug 12, 2019, at 6:34 AM, Alan Bateman wrote: > > On 09/08/2019 17:18, Brian Burkhalter wrote: >> https://bugs.openjdk.java.net/browse/JDK-6937320 >> http://cr.openjdk.java.net/~bpb/6937320/webrev.00/ >> >> The failure addressed here occurred on Linux when trying to obtain the FileStore of ?/? which on some misconfigured systems was ?rootfs? instead of the actual device due to duplicate entries for ?/? in /proc/mounts with the rootfs entry being first. The change is to attempt to recognize an actual device path, e.g., /dev/sda1, in the entry and to use the first non-path entry, e.g., rootfs, only if a path-like entry was not encountered first. >> > Do you have examples, or maybe any references, to how these mount entries are setup? I think we need these before deciding if changes are needed. That this can happen is corroborated by the results of web searches but I don?t recall seeing it myself in actuality. If its occurrence looks too rare then maybe is should just be resolved as NAI. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Aug 12 23:05:38 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 12 Aug 2019 16:05:38 -0700 Subject: 8225461: (fs) Investigate filtering duplicates out of Iterable returned by java.nio.file.FileSystem::getFileStores In-Reply-To: <47D28DD6-B089-471E-A2D3-C13362ED8B3F@oracle.com> References: <9d73d427-f83a-cedc-45cb-1c6fa1299306@oracle.com> <13876EA1-D51B-4B67-ADED-C7174D46235F@oracle.com> <47D28DD6-B089-471E-A2D3-C13362ED8B3F@oracle.com> Message-ID: <6B4F0C45-AE9B-476F-964D-E4423AAF9153@oracle.com> > On Aug 9, 2019, at 11:54 AM, Brian Burkhalter wrote: > >> On Aug 9, 2019, at 11:49 AM, Bernd Eckenfels > wrote: >> >> I wonder if for the df case you should word-split the mountpoints, assuming that if duplicate lines are possible they might differ in any of the other columns which are not relevant for the test. (And i wonder if this as well as the /proc/mounts Support Blanks in the path) > > That might be a good idea. Here is a modified version [1] which uses only the device/filesystem and mount point fields of the df and /proc/mounts lines. Thanks, Brian [1] http://cr.openjdk.java.net/~bpb/8225461/webrev.02/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Aug 13 07:02:46 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 13 Aug 2019 08:02:46 +0100 Subject: 6937320: (fs) Path getFileStore().type() does not return correct type of file store on Linux In-Reply-To: <27070CD6-DE3F-4F4A-B24E-995B4C5BBC56@oracle.com> References: <9A8E7D3B-9085-4F83-8AEE-B56263BE19FA@oracle.com> <27070CD6-DE3F-4F4A-B24E-995B4C5BBC56@oracle.com> Message-ID: On 12/08/2019 17:01, Brian Burkhalter wrote: > : > > That this can happen is corroborated by the results of web searches > but I don?t recall seeing it myself in actuality. If its occurrence > looks too rare then maybe is should just be resolved as NAI. > I don't think it's possible to review the changes without some examples as to how these mounts are exposed in the mount tables. Can we get that from the submitters and attach it to the bug reports? I think that has to be starting point before discussing any changes. -Alan From Alan.Bateman at oracle.com Fri Aug 16 10:39:02 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 16 Aug 2019 11:39:02 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <9b546ca7-b664-a681-7ee6-f074d49081b4@redhat.com> <56ae56b8-5170-cb45-17eb-d3a326f7560e@oracle.com> <31edb02d-b178-6e83-da02-f98e9d67d28e@redhat.com> Message-ID: <58637bf9-fb6b-fb67-01ee-1452d8b95190@oracle.com> On 15/08/2019 12:53, Andrew Dinn wrote: > Hi Alan, > > Any further input on this patch forthcoming? > I think the main changes since I looked at it previously have been in the tests. On non-Linux platforms, FileChannel.map should throw UOE when invoked with a mode map of READ_ONLY_SYNC or READ_WRITE_SYNC so I think we need a test for that. MapFail seems fragile. If you add @modules java.base/jdk.internal.misc to the test description then it could use Unsafe::isWritebackEnabled and I think would simplify the test and checks. It would mean it could run on all platforms. Also "MapFail" is probably too general a name for a test in MBB because its specific to the extended map modes, not a general test of map failing. In passing, MappedByteBuffer load/isLoaded check the fd value before isSync, can force() do the same? Also the @return on the private isSync method is very wordy and I don't think needs to duplicate the method description. -Alan From franta-java at frantovo.cz Mon Aug 19 08:34:19 2019 From: franta-java at frantovo.cz (=?UTF-8?B?RnJhbnRpxaFlayBLdcSNZXJh?=) Date: Mon, 19 Aug 2019 10:34:19 +0200 Subject: Unix domain sockets (UDS, AF_UNIX) in System.inheritedChannel() and elsewhere in Java In-Reply-To: <85cf2869-4fa6-a246-d8f5-ce56890f37ad@oracle.com> References: <5e31f8ab-9007-a501-1704-935e79f0133f@frantovo.cz> <4b57b5c2-afea-3ea9-73e2-0e3913ca6ef7@oracle.com> <0aa6e746-0a29-53ba-27dd-e36ad71e3b26@frantovo.cz> <75b60535-ab55-2e3c-d283-a36b78e4175c@frantovo.cz> <85cf2869-4fa6-a246-d8f5-ce56890f37ad@oracle.com> Message-ID: Dne 23. 07. 19 v 17:46 Alan Bateman napsal(a): > On 22/07/2019 09:16, Franti?ek Ku?era wrote: >> >> It is a stream or a datagram socket ? just from different domain >> (UNIX instead of INET or INET6) ? but they work the same way. So yes, >> when I create a stream or datagram unix domain socket e.g. in systemd >> (parent process), I can inherit it in Java and get >> ServerSocketChannel or DatagramChannel from System.inheritedChannel() >> and it really works. Recently, I wrote an article about it: >> https://blog.frantovo.cz/c/372/ ? it is in Czech, but code and config >> examples should be readable to anyone. >> >> : >> >> BTW: the inheritedChannel() JavaDoc says: >> >> > In addition to the network-oriented channels described, this method >> may return other kinds of channels in the future. >> > Yes, we added this method in Java SE 5 with a view to supporting other > channel types (e.g. console) in the future. > > I'll get back to you on the other points soon but I was curious about > your observation that the inheritedChannel returns a channels with > "garbage" local/remote addresses when fd 0 is a Unix domain socket. > The bug is that the probe is missing a call to getsockname that it > needs to check if the family is AF_INET or AF_INET6. So we should > minimally fix that. > > -Alan Hello, can anyone tell me how to continue? I can change the code (the patch was only proof-of-concept, fast hack) or change the design (e.g. creating a new module instead of patching existing classes) but before I spend more time with this task, I need to know, whether there is interest to have AF_UNIX (AF_LOCAL) support in the OpenJDK (see the first e-mail in this thread for what I mean by "support"). Franta From chris.hegarty at oracle.com Mon Aug 19 10:47:14 2019 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 19 Aug 2019 11:47:14 +0100 Subject: Unix domain sockets (UDS, AF_UNIX) in System.inheritedChannel() and elsewhere in Java In-Reply-To: References: <5e31f8ab-9007-a501-1704-935e79f0133f@frantovo.cz> <4b57b5c2-afea-3ea9-73e2-0e3913ca6ef7@oracle.com> <0aa6e746-0a29-53ba-27dd-e36ad71e3b26@frantovo.cz> <75b60535-ab55-2e3c-d283-a36b78e4175c@frantovo.cz> <85cf2869-4fa6-a246-d8f5-ce56890f37ad@oracle.com> Message-ID: Franta, > On 19 Aug 2019, at 09:34, Franti?ek Ku?era wrote: > > ... > can anyone tell me how to continue? I can change the code (the patch was only proof-of-concept, fast hack) or change the design (e.g. creating a new module instead of patching existing classes) but before I spend more time with this task, I need to know, whether there is interest to have AF_UNIX (AF_LOCAL) support in the OpenJDK (see the first e-mail in this thread for what I mean by "support?). From my perspective, it would be reasonable to continue to investigate the possibility of supporting Unix domain sockets in the JDK. Such a feature would likely need a JEP [1]. If I'm not mistaken, then this is your first contribution to OpenJDK ( it's a pretty significant/large one to start with! ). You will need to find a sponsor ( a committer on the JDK project ), that is both motivated to help you drive the design and implementation, as well as having the time available to do so ( I, unfortunately, do not ). While this feature does, on the face of it, seem reasonable, in terms of priority, it does not bubble up the list of other things that we want to do, like; providing support for the newer URI/IRI, improved IPv6-only support, RDMA Network Sockets, etc. -Chris. [1] https://openjdk.java.net/jeps/0 From vkempik at azul.com Mon Aug 19 11:22:21 2019 From: vkempik at azul.com (Vladimir Kempik) Date: Mon, 19 Aug 2019 11:22:21 +0000 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent Message-ID: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> Hello Please review the suggested fix for bug JDK-8229872 Files.getFileStore() will fail if there is an entry in /proc/mounts which is greater than 1024 bytes length This happens because we allocate 1024 bytes size buffer on stack for use with getmntent_r. A possible way is to use getmntent instead, it will allocated 4kb buffer on glibc and ?as much as needed? on musl (https://www.openwall.com/lists/musl/2016/09/08/1 ) But getmntent is dubbed "MT-Unsafe race:mntentbuf locale? I have tested this fix on openjdk8u222 and it can parse long entries fine. Any suggestions? Thanks, Vladimir The bug: https://bugs.openjdk.java.net/browse/JDK-8229872 The webrev: http://cr.openjdk.java.net/~vkempik/8229872/webrev.01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From adinn at redhat.com Mon Aug 19 11:29:32 2019 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 19 Aug 2019 12:29:32 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <58637bf9-fb6b-fb67-01ee-1452d8b95190@oracle.com> References: <9b546ca7-b664-a681-7ee6-f074d49081b4@redhat.com> <56ae56b8-5170-cb45-17eb-d3a326f7560e@oracle.com> <31edb02d-b178-6e83-da02-f98e9d67d28e@redhat.com> <58637bf9-fb6b-fb67-01ee-1452d8b95190@oracle.com> Message-ID: Hi Alan, Thanks for looking at the patch again. I think I have addressed all your concerns (comments inline). Webrev and retest results at the end. On 16/08/2019 11:39, Alan Bateman wrote: > I think the main changes since I looked at it previously have been in > the tests. That's mostly it. I did also fix a problem that was breaking build of x86_32. I also ensured that MAP_SYNC maps fail as expected on that arch. > On non-Linux platforms, FileChannel.map should throw UOE when invoked > with a mode map of READ_ONLY_SYNC or READ_WRITE_SYNC so I think we need > a test for that. > > MapFail seems fragile. If you add @modules java.base/jdk.internal.misc > to the test description then it could use Unsafe::isWritebackEnabled and > I think would simplify the test and checks. It would mean it could run > on all platforms. Also "MapFail" is probably too general a name for a > test in MBB because its specific to the extended map modes, not a > general test of map failing. I renamed the test to MapSyncFail and modified it to run without restriction to a specific os or cpu. I also generalized it to run twice with a boolean arg which selects mode READ_ONLY_SYNC on the first run and READ_WRITE_SYNC on the second one. The logic of the test is now to expect 1) IOException if Unsafe.isWriteBackEnabled -> true 2) UnsupportedOperationException if Unsafe.isWriteBackEnabled -> false If the wrong exception or neither exception is thrown the test fails. Case 1 currently only applies for x86_64. Case 2 applies for all other architectures. > In passing, MappedByteBuffer load/isLoaded check the fd value before > isSync, can force() do the same? Also the @return on the private isSync > method is very wordy and I don't think needs to duplicate the method > description. Sure, I have modified force() to do that check first. Of course, that means that force(int, int) is going to repeat the same test -- it has to because it may be called direct without going via force(). However, that's not really a problem since the compiler should elide the repeated check. I also shortened the text following the @return annotation as requested. Updated webrev: http://cr.openjdk.java.net/~adinn/8224974/webrev.12 Testing: Test PmemTest: passes as expected on x86_64 (only arch for which DAX file system is available) fails to pass as expected on aarch64 and x86_32 (however, this case is covered by the next test) Test MapSyncFail: passes with expected exceptions on Linux for x86_64 (IOException), aarch64 and x86_32 (UnsupportedOperationException). not tested on other arch/OS combinations (I have no access to the necessary kit). Red Hat MW tests: All still passing successfully submit test: still in progress Is it ok to push if the submit test comes back clean? regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From fweimer at redhat.com Mon Aug 19 11:56:08 2019 From: fweimer at redhat.com (Florian Weimer) Date: Mon, 19 Aug 2019 13:56:08 +0200 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> (Vladimir Kempik's message of "Mon, 19 Aug 2019 11:22:21 +0000") References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> Message-ID: <87ftlxtllz.fsf@oldenburg2.str.redhat.com> * Vladimir Kempik: > Please review the suggested fix for bug JDK-8229872 > > Files.getFileStore() will fail if there is an entry in /proc/mounts which is greater than 1024 bytes length > This happens because we allocate 1024 bytes size buffer on stack for use with getmntent_r. > > A possible way is to use getmntent instead, it will allocated 4kb buffer on glibc and ?as much as needed? on musl > (https://www.openwall.com/lists/musl/2016/09/08/1) > But getmntent is dubbed "MT-Unsafe race:mntentbuf locale? > > I have tested this fix on openjdk8u222 and it can parse long entries fine. I think the bug still persists because a mount path can be up to 4095 bytes long, and it can be expanded to four times that using octal escapes. getmntent really isn't thread-safe in the current glibc implementation, so I don't think you can use that. I think the easiest solution will be to parse /proc/mounts directly, from Java code. The file format is expected to be stable and quite simple. The code probably won't be much longer than for parsing struct mntent. 8-/ In the future, there might be an RFE to ignore certain entries (e.g., those installed by autofs), but the kernel doesn't currently provide the flag required for that. Thanks, Florian From Alan.Bateman at oracle.com Mon Aug 19 12:46:36 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 19 Aug 2019 13:46:36 +0100 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <87ftlxtllz.fsf@oldenburg2.str.redhat.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> Message-ID: <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> On 19/08/2019 12:56, Florian Weimer wrote: > * Vladimir Kempik: > >> : >> >> A possible way is to use getmntent instead, it will allocated 4kb buffer on glibc and ?as much as needed? on musl >> (https://www.openwall.com/lists/musl/2016/09/08/1) >> But getmntent is dubbed "MT-Unsafe race:mntentbuf locale? >> >> Is errno set to something useful when getmntent_r fails? Just wondering if it can be used to allocate a larger buffer. -Alan From fweimer at redhat.com Mon Aug 19 12:54:40 2019 From: fweimer at redhat.com (Florian Weimer) Date: Mon, 19 Aug 2019 14:54:40 +0200 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> (Alan Bateman's message of "Mon, 19 Aug 2019 13:46:36 +0100") References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> Message-ID: <87v9uts4bz.fsf@oldenburg2.str.redhat.com> * Alan Bateman: > On 19/08/2019 12:56, Florian Weimer wrote: >> * Vladimir Kempik: >> >>> : >>> >>> A possible way is to use getmntent instead, it will allocated 4kb buffer on glibc and ?as much as needed? on musl >>> (https://www.openwall.com/lists/musl/2016/09/08/1) >>> But getmntent is dubbed "MT-Unsafe race:mntentbuf locale? >>> >>> > Is errno set to something useful when getmntent_r fails? Just > wondering if it can be used to allocate a larger buffer. getmntent_r currently ignores overlong lines. One problem is that the interface is fairly broken: Reading a line consumes it from the stdio stream, and one can determine the required buffer size only after doing that. This means that the input has to be seekable, so that the file pointer can be restored to the original value in case the supplied buffer is too small. /proc/mounts is seekable, so it could work, but there have been quite a few buffer boundary issues in the kernel's procfs code, so I'm a bit wary of making this change. getmntent is easier to fix because glibc can manage the buffer allocation internally and make it thread-local. But it will take years until a potential fix reaches most distributions, and until that happens, using getmntent makes things potentially worse. Thanks, Florian From vkempik at azul.com Mon Aug 19 13:01:42 2019 From: vkempik at azul.com (Vladimir Kempik) Date: Mon, 19 Aug 2019 13:01:42 +0000 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <87v9uts4bz.fsf@oldenburg2.str.redhat.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> Message-ID: <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> Maybe it can be done it this way: in jni?s Java_sun_nio_fs_LinuxNativeDispatcher_setmntent0 go thru file line by line ( with getline) and find longest line, then rewind to file begin. Thanks, Vladimir > 19 ???. 2019 ?., ? 15:54, Florian Weimer ???????(?): > > * Alan Bateman: > >> On 19/08/2019 12:56, Florian Weimer wrote: >>> * Vladimir Kempik: >>> >>>> : >>>> >>>> A possible way is to use getmntent instead, it will allocated 4kb buffer on glibc and ?as much as needed? on musl >>>> (https://www.openwall.com/lists/musl/2016/09/08/1) >>>> But getmntent is dubbed "MT-Unsafe race:mntentbuf locale? >>>> >>>> > >> Is errno set to something useful when getmntent_r fails? Just >> wondering if it can be used to allocate a larger buffer. > > getmntent_r currently ignores overlong lines. > > One problem is that the interface is fairly broken: Reading a line > consumes it from the stdio stream, and one can determine the required > buffer size only after doing that. This means that the input has to be > seekable, so that the file pointer can be restored to the original value > in case the supplied buffer is too small. /proc/mounts is seekable, so > it could work, but there have been quite a few buffer boundary issues in > the kernel's procfs code, so I'm a bit wary of making this change. > > getmntent is easier to fix because glibc can manage the buffer > allocation internally and make it thread-local. But it will take years > until a potential fix reaches most distributions, and until that > happens, using getmntent makes things potentially worse. > > Thanks, > Florian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From fweimer at redhat.com Mon Aug 19 13:18:01 2019 From: fweimer at redhat.com (Florian Weimer) Date: Mon, 19 Aug 2019 15:18:01 +0200 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> (Vladimir Kempik's message of "Mon, 19 Aug 2019 13:01:42 +0000") References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> Message-ID: <87lfvps392.fsf@oldenburg2.str.redhat.com> * Vladimir Kempik: > Maybe it can be done it this way: > > in jni?s Java_sun_nio_fs_LinuxNativeDispatcher_setmntent0 > > go thru file line by line ( with getline) and find longest line, then rewind to file begin. Interesting idea. It still has a small race condition. But it might be the best compromise between amount of changes required and gains in robustness. Thanks, Florian From Alan.Bateman at oracle.com Mon Aug 19 14:36:20 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 19 Aug 2019 15:36:20 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <9b546ca7-b664-a681-7ee6-f074d49081b4@redhat.com> <56ae56b8-5170-cb45-17eb-d3a326f7560e@oracle.com> <31edb02d-b178-6e83-da02-f98e9d67d28e@redhat.com> <58637bf9-fb6b-fb67-01ee-1452d8b95190@oracle.com> Message-ID: On 19/08/2019 12:29, Andrew Dinn wrote: > : > I renamed the test to MapSyncFail and modified it to run without > restriction to a specific os or cpu. > > I also generalized it to run twice with a boolean arg which selects mode > READ_ONLY_SYNC on the first run and READ_WRITE_SYNC on the second one. > > The logic of the test is now to expect > > 1) IOException if Unsafe.isWriteBackEnabled -> true > 2) UnsupportedOperationException if Unsafe.isWriteBackEnabled -> false > > If the wrong exception or neither exception is thrown the test fails. > > Case 1 currently only applies for x86_64. > Case 2 applies for all other architectures. Thanks, this looks must better and avoids checking the architecture and exception messages. I think webrev.12 looks good; I don't have any other comments. -Alan From dmitry.chuyko at bell-sw.com Mon Aug 19 18:38:55 2019 From: dmitry.chuyko at bell-sw.com (Dmitry Chuyko) Date: Mon, 19 Aug 2019 21:38:55 +0300 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <9b546ca7-b664-a681-7ee6-f074d49081b4@redhat.com> <56ae56b8-5170-cb45-17eb-d3a326f7560e@oracle.com> <31edb02d-b178-6e83-da02-f98e9d67d28e@redhat.com> <58637bf9-fb6b-fb67-01ee-1452d8b95190@oracle.com> Message-ID: Hi Andrew, Just a minor style thing in MapSyncFail test: can "true" and "false" (the mode) be "READ_WRITE_SYNC" and "READ_ONLY_SYNC" instead? -Dmitry On 8/19/19 2:29 PM, Andrew Dinn wrote: > Hi Alan, > > ................... > I renamed the test to MapSyncFail and modified it to run without > restriction to a specific os or cpu. > > I also generalized it to run twice with a boolean arg which selects mode > READ_ONLY_SYNC on the first run and READ_WRITE_SYNC on the second one. > > The logic of the test is now to expect > > 1) IOException if Unsafe.isWriteBackEnabled -> true > 2) UnsupportedOperationException if Unsafe.isWriteBackEnabled -> false > > If the wrong exception or neither exception is thrown the test fails. > > Case 1 currently only applies for x86_64. > Case 2 applies for all other architectures. > >> In passing, MappedByteBuffer load/isLoaded check the fd value before >> isSync, can force() do the same? Also the @return on the private isSync >> method is very wordy and I don't think needs to duplicate the method >> description. > Sure, I have modified force() to do that check first. > > Of course, that means that force(int, int) is going to repeat the same > test -- it has to because it may be called direct without going via force(). > > However, that's not really a problem since the compiler should elide the > repeated check. > > I also shortened the text following the @return annotation as requested. > > Updated webrev: > > http://cr.openjdk.java.net/~adinn/8224974/webrev.12 > > Testing: > > Test PmemTest: > passes as expected on x86_64 (only arch for which DAX file system is > available) > fails to pass as expected on aarch64 and x86_32 (however, this case is > covered by the next test) > > Test MapSyncFail: > passes with expected exceptions on Linux for x86_64 (IOException), > aarch64 and x86_32 (UnsupportedOperationException). > not tested on other arch/OS combinations (I have no access to the > necessary kit). > > Red Hat MW tests: > All still passing successfully > > submit test: > still in progress > > Is it ok to push if the submit test comes back clean? > > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From lance.andersen at oracle.com Mon Aug 19 20:48:18 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Mon, 19 Aug 2019 13:48:18 -0700 (PDT) Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method Message-ID: Hi The following patch for JDK-8229887 addresses an issue introduced by the performance fix in https://bugs.openjdk.java.net/browse/JDK-8222532 resulting in an error when trying to replace an entry in a ZIP/JAR file that was added with no-compression. The webrev can be found at http://cr.openjdk.java.net/~lancea/8229887/webrev.00/index.html Mach5 tier1 - tier3 runs are all clean with this fix. Best Lance Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From claes.redestad at oracle.com Mon Aug 19 21:13:53 2019 From: claes.redestad at oracle.com (Claes Redestad) Date: Mon, 19 Aug 2019 23:13:53 +0200 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: References: Message-ID: Hi Lance, On 2019-08-19 22:48, Lance Andersen wrote: > http://cr.openjdk.java.net/~lancea/8229887/webrev.00/index.html test and fix looks good. /Claes From martinrb at google.com Mon Aug 19 22:55:48 2019 From: martinrb at google.com (Martin Buchholz) Date: Mon, 19 Aug 2019 15:55:48 -0700 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: References: Message-ID: (not a review, but) For many years I've been helping to maintain java.util.zip, but I've mostly ignored zipfs, and zipfs spent years as a "demo", so it is probably undertested, and it would help if existing tests for e.g. ZipFile were extended to also cover zipfs when that was a natural thing to do. On Mon, Aug 19, 2019 at 1:49 PM Lance Andersen wrote: > Hi > > The following patch for JDK-8229887 > addresses an issue > introduced by the performance fix in > https://bugs.openjdk.java.net/browse/JDK-8222532 resulting in an error > when trying to replace an entry in a ZIP/JAR file that was added with > no-compression. > > The webrev can be found at > http://cr.openjdk.java.net/~lancea/8229887/webrev.00/index.html > > Mach5 tier1 - tier3 runs are all clean with this fix. > > > Best > Lance > > > > > > > Lance Andersen| > Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance.andersen at oracle.com Mon Aug 19 23:02:39 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Mon, 19 Aug 2019 19:02:39 -0400 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: References: Message-ID: <6B4E23DB-4245-4B94-A640-61B171CEF2D6@oracle.com> Thank you Martin. I agree for the need to continue to improve coverage. It has gotten somewhat better but definitely on the todo list. Best Lance > On Aug 19, 2019, at 6:55 PM, Martin Buchholz wrote: > > (not a review, but) > For many years I've been helping to maintain java.util.zip, but I've mostly ignored zipfs, and zipfs spent years as a "demo", so it is probably undertested, and it would help if existing tests for e.g. ZipFile were extended to also cover zipfs when that was a natural thing to do. > > On Mon, Aug 19, 2019 at 1:49 PM Lance Andersen > wrote: > Hi > > The following patch for JDK-8229887 addresses an issue introduced by the performance fix in https://bugs.openjdk.java.net/browse/JDK-8222532 resulting in an error when trying to replace an entry in a ZIP/JAR file that was added with no-compression. > > The webrev can be found at http://cr.openjdk.java.net/~lancea/8229887/webrev.00/index.html > > Mach5 tier1 - tier3 runs are all clean with this fix. > > > Best > Lance > > > > > > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From adinn at redhat.com Tue Aug 20 08:02:14 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 20 Aug 2019 09:02:14 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <9b546ca7-b664-a681-7ee6-f074d49081b4@redhat.com> <56ae56b8-5170-cb45-17eb-d3a326f7560e@oracle.com> <31edb02d-b178-6e83-da02-f98e9d67d28e@redhat.com> <58637bf9-fb6b-fb67-01ee-1452d8b95190@oracle.com> Message-ID: <9071c16a-6ef1-fe58-8233-764c3ab0e1f1@redhat.com> On 19/08/2019 19:38, Dmitry Chuyko wrote: > Just a minor style thing in MapSyncFail test: can "true" and "false" > (the mode) be "READ_WRITE_SYNC" and "READ_ONLY_SYNC" instead? Are you referring to the arguments passed on the command line? If so then the answer is clearly yes. However, I'm not sure why you consider this change important and/or an improvement? Do you feel that the code which processes the argument to compute the Mode setting is obscure? Or perhaps you are referring to something other than the command line arguments? I /do/ need to fix the typo in the test where is says "expected true of false as an argument" Obviously that should say "true *or* false". regards, Andrew Dinn ----------- From adinn at redhat.com Tue Aug 20 09:36:16 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 20 Aug 2019 10:36:16 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <9b546ca7-b664-a681-7ee6-f074d49081b4@redhat.com> <56ae56b8-5170-cb45-17eb-d3a326f7560e@oracle.com> <31edb02d-b178-6e83-da02-f98e9d67d28e@redhat.com> <58637bf9-fb6b-fb67-01ee-1452d8b95190@oracle.com> Message-ID: On 19/08/2019 15:36, Alan Bateman wrote: > I think webrev.12 looks good; I don't have any other comments. Thanks, Alan. I just pushed the patch for the JEP implementation. (Hallelujah!). Thanks for all your help. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From dmitry.chuyko at bell-sw.com Tue Aug 20 11:16:18 2019 From: dmitry.chuyko at bell-sw.com (Dmitry Chuyko) Date: Tue, 20 Aug 2019 14:16:18 +0300 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <9071c16a-6ef1-fe58-8233-764c3ab0e1f1@redhat.com> References: <9b546ca7-b664-a681-7ee6-f074d49081b4@redhat.com> <56ae56b8-5170-cb45-17eb-d3a326f7560e@oracle.com> <31edb02d-b178-6e83-da02-f98e9d67d28e@redhat.com> <58637bf9-fb6b-fb67-01ee-1452d8b95190@oracle.com> <9071c16a-6ef1-fe58-8233-764c3ab0e1f1@redhat.com> Message-ID: On 8/20/19 11:02 AM, Andrew Dinn wrote: > On 19/08/2019 19:38, Dmitry Chuyko wrote: >> Just a minor style thing in MapSyncFail test: can "true" and "false" >> (the mode) be "READ_WRITE_SYNC" and "READ_ONLY_SYNC" instead? > Are you referring to the arguments passed on the command line? If so > then the answer is clearly yes. Yes, exactly that. Trivial change like a string switch wouldn't bloat the code much. Anyway, probably it was already a time to push changes :-) Latest version of MapSyncFail covers exception path then writeback is not enabled so I think we don't need additional work here either. Thanks for a good job! -Dmitry > > .......... > > regards, > > > Andrew Dinn > ----------- > From Alan.Bateman at oracle.com Tue Aug 20 11:34:16 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Aug 2019 12:34:16 +0100 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: References: Message-ID: On 19/08/2019 21:48, Lance Andersen wrote: > Hi > > The following patch for JDK-8229887 > ?addresses an issue > introduced by the performance fix in > https://bugs.openjdk.java.net/browse/JDK-8222532 resulting in an error > when trying to replace an entry in a ZIP/JAR file that was added with > no-compression. > > The webrev can be found at > http://cr.openjdk.java.net/~lancea/8229887/webrev.00/index.html > The update to ZipFileSystem looks good. I think the test needs a bit of work and needs to test updating zip files where all entries are stored, all entries are deflated, and a mix of the two. I've created a test that does this if you need it. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From vkempik at azul.com Tue Aug 20 13:43:49 2019 From: vkempik at azul.com (Vladimir Kempik) Date: Tue, 20 Aug 2019 13:43:49 +0000 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <87lfvps392.fsf@oldenburg2.str.redhat.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> Message-ID: <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> Hello I?m planning to sync setmntent/getmntent on some object via MonitorEnter/Exit to prevent race condition. But where to get some jobject ? Should I pass it from caller (java method), but then It needs to be some static object. Or should I create my own on first usage of setmntent ? Thanks, Vladimir. > 19 ???. 2019 ?., ? 16:18, Florian Weimer ???????(?): > > * Vladimir Kempik: > >> Maybe it can be done it this way: >> >> in jni?s Java_sun_nio_fs_LinuxNativeDispatcher_setmntent0 >> >> go thru file line by line ( with getline) and find longest line, then rewind to file begin. > > Interesting idea. It still has a small race condition. But it might be > the best compromise between amount of changes required and gains in > robustness. > > Thanks, > Florian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From Alan.Bateman at oracle.com Tue Aug 20 13:47:57 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Aug 2019 14:47:57 +0100 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> Message-ID: <4933c9f0-7a73-e215-421e-e4ea30f5c810@oracle.com> On 20/08/2019 14:43, Vladimir Kempik wrote: > Hello > > I?m planning to sync setmntent/getmntent on some object via MonitorEnter/Exit to prevent race condition. > But where to get some jobject ? > Should I pass it from caller (java method), but then It needs to be some static object. > Or should I create my own on first usage of setmntent ? If you are talking about parsing the file then you shouldn't need any code to do this. In this case, we try to keep the JNI methods to be as simple as possible, usually just to wrap a system or lib call. -Alan. From vkempik at azul.com Tue Aug 20 14:00:40 2019 From: vkempik at azul.com (Vladimir Kempik) Date: Tue, 20 Aug 2019 14:00:40 +0000 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <4933c9f0-7a73-e215-421e-e4ea30f5c810@oracle.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <4933c9f0-7a73-e215-421e-e4ea30f5c810@oracle.com> Message-ID: Hello Sorry I wasn?t clear. Parsing will be simple. (dup, getline, close) The issue is to synchronize result (max line width, and realloced buffer (if needed)) between setmntent0 and getmntent. Thanks, Vladimir > 20 ???. 2019 ?., ? 16:47, Alan Bateman ???????(?): > > On 20/08/2019 14:43, Vladimir Kempik wrote: >> Hello >> >> I?m planning to sync setmntent/getmntent on some object via MonitorEnter/Exit to prevent race condition. >> But where to get some jobject ? >> Should I pass it from caller (java method), but then It needs to be some static object. >> Or should I create my own on first usage of setmntent ? > If you are talking about parsing the file then you shouldn't need any code to do this. In this case, we try to keep the JNI methods to be as simple as possible, usually just to wrap a system or lib call. > > -Alan. From fweimer at redhat.com Tue Aug 20 14:38:44 2019 From: fweimer at redhat.com (Florian Weimer) Date: Tue, 20 Aug 2019 16:38:44 +0200 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> (Vladimir Kempik's message of "Tue, 20 Aug 2019 13:43:49 +0000") References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> Message-ID: <8736hvoqa3.fsf@oldenburg2.str.redhat.com> * Vladimir Kempik: > I?m planning to sync setmntent/getmntent on some object via MonitorEnter/Exit to prevent race condition. > But where to get some jobject ? > Should I pass it from caller (java method), but then It needs to be some static object. > Or should I create my own on first usage of setmntent ? The new race condition is with other processes (not even Java processes) which do mounts. There is no way to avoid it reliably. I still recommend to implement the file parsing in Java. It's more code than what's there today only because all of it would currently be Linux-specific. Otherwise I expect to be it a simplification. Thanks, Florian From ivan.gerasimov at oracle.com Tue Aug 20 17:38:49 2019 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Tue, 20 Aug 2019 10:38:49 -0700 Subject: RFR 8211360 : Change #if DEF to #if defined(DEF) Message-ID: <0d38c07a-e697-49c6-dfc0-938a8ffa7820@oracle.com> Hello! It's a followup for JDK-8211146. With that fix several C-preprocessor statements of form #elif __linux__ were changed to more accurate #elif defined(__linux__). grep found a few more occurrences of the same pattern, which also would better be cleaned up. BUGURL: https://bugs.openjdk.java.net/browse/JDK-8211360 WEBREV: http://cr.openjdk.java.net/~igerasim/8211360/00/webrev/ Mach5 control build/testing went fine on all platforms. Would you please help review? -- With kind regards, Ivan Gerasimov From brian.burkhalter at oracle.com Tue Aug 20 17:56:30 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 20 Aug 2019 10:56:30 -0700 Subject: RFR 8211360 : Change #if DEF to #if defined(DEF) In-Reply-To: <0d38c07a-e697-49c6-dfc0-938a8ffa7820@oracle.com> References: <0d38c07a-e697-49c6-dfc0-938a8ffa7820@oracle.com> Message-ID: <899FC93B-95C8-4E53-B1EA-E4C19EE3F58E@oracle.com> Hello Ivan, Looks fine. Brian > On Aug 20, 2019, at 10:38 AM, Ivan Gerasimov wrote: > > It's a followup for JDK-8211146. > > With that fix several C-preprocessor statements of form #elif __linux__ were changed to more accurate #elif defined(__linux__). > > grep found a few more occurrences of the same pattern, which also would better be cleaned up. > > BUGURL: https://bugs.openjdk.java.net/browse/JDK-8211360 > WEBREV: http://cr.openjdk.java.net/~igerasim/8211360/00/webrev/ > > Mach5 control build/testing went fine on all platforms. > Would you please help review? -------------- next part -------------- An HTML attachment was scrubbed... URL: From vkempik at azul.com Tue Aug 20 19:31:01 2019 From: vkempik at azul.com (Vladimir Kempik) Date: Tue, 20 Aug 2019 19:31:01 +0000 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <8736hvoqa3.fsf@oldenburg2.str.redhat.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> Message-ID: <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> Hello after reading manpage for getline I think I have found some elegant solution - to let getline manage size of buffer. Please review v2 of patch here http://cr.openjdk.java.net/~vkempik/8229872/webrev.02/ I?m still worried about syncing maxlinelen and linebuffer between Java_sun_nio_fs_LinuxNativeDispatcher_setmntent0 and Java_sun_nio_fs_LinuxNativeDispatcher_getmntent, if they are run from different threads. Thanks, Vladimir > 20 ???. 2019 ?., ? 17:38, Florian Weimer ???????(?): > > * Vladimir Kempik: > >> I?m planning to sync setmntent/getmntent on some object via MonitorEnter/Exit to prevent race condition. >> But where to get some jobject ? >> Should I pass it from caller (java method), but then It needs to be some static object. >> Or should I create my own on first usage of setmntent ? > > The new race condition is with other processes (not even Java processes) > which do mounts. There is no way to avoid it reliably. > > I still recommend to implement the file parsing in Java. It's more code > than what's there today only because all of it would currently be > Linux-specific. Otherwise I expect to be it a simplification. > > Thanks, > Florian From brian.burkhalter at oracle.com Tue Aug 20 19:40:43 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 20 Aug 2019 12:40:43 -0700 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> Message-ID: Hello Vladimir, L2: The newer copyright year should be changed to 2019. L170: The line is too long. L236: Before this line I think you should free() linebuffer. Thanks, Brian > On Aug 20, 2019, at 12:31 PM, Vladimir Kempik wrote: > > after reading manpage for getline I think I have found some elegant solution - to let getline manage size of buffer. > > Please review v2 of patch here http://cr.openjdk.java.net/~vkempik/8229872/webrev.02/ > > I?m still worried about syncing maxlinelen and linebuffer between Java_sun_nio_fs_LinuxNativeDispatcher_setmntent0 and Java_sun_nio_fs_LinuxNativeDispatcher_getmntent, if they are run from different threads. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vkempik at azul.com Tue Aug 20 19:46:21 2019 From: vkempik at azul.com (Vladimir Kempik) Date: Tue, 20 Aug 2019 19:46:21 +0000 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> Message-ID: Hello about L236, I really think I should not linebuffer will leave throughtout app lifetime and will be realloced only when needed. It?s kind of cache now, for use by getmntent_r. if I free linebuffer and set it to NULL, then I will have to set maxlinelen to 0 as well, otherwise getline will fail. This means the app will have to recalc maxline ( and do some reallocs to linebuffer) every time setmntent0 is called. So I believe it?s better to let linebuffer leave, it?s not memory leak as we are tracking it and using it still. Thanks, Vladimir. > 20 ???. 2019 ?., ? 22:40, Brian Burkhalter ???????(?): > > Hello Vladimir, > > L2: The newer copyright year should be changed to 2019. > L170: The line is too long. > L236: Before this line I think you should free() linebuffer. > > Thanks, > > Brian > >> On Aug 20, 2019, at 12:31 PM, Vladimir Kempik > wrote: >> >> after reading manpage for getline I think I have found some elegant solution - to let getline manage size of buffer. >> >> Please review v2 of patch here http://cr.openjdk.java.net/~vkempik/8229872/webrev.02/ >> >> I?m still worried about syncing maxlinelen and linebuffer between Java_sun_nio_fs_LinuxNativeDispatcher_setmntent0 and Java_sun_nio_fs_LinuxNativeDispatcher_getmntent, if they are run from different threads. > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From vkempik at azul.com Tue Aug 20 19:50:06 2019 From: vkempik at azul.com (Vladimir Kempik) Date: Tue, 20 Aug 2019 19:50:06 +0000 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> Message-ID: Sorry, live, not leave, x2. 20 ???. 2019 ?., ? 22:46, Vladimir Kempik > ???????(?): Hello about L236, I really think I should not linebuffer will leave throughtout app lifetime and will be realloced only when needed. It?s kind of cache now, for use by getmntent_r. if I free linebuffer and set it to NULL, then I will have to set maxlinelen to 0 as well, otherwise getline will fail. This means the app will have to recalc maxline ( and do some reallocs to linebuffer) every time setmntent0 is called. So I believe it?s better to let linebuffer leave, it?s not memory leak as we are tracking it and using it still. Thanks, Vladimir. 20 ???. 2019 ?., ? 22:40, Brian Burkhalter > ???????(?): Hello Vladimir, L2: The newer copyright year should be changed to 2019. L170: The line is too long. L236: Before this line I think you should free() linebuffer. Thanks, Brian On Aug 20, 2019, at 12:31 PM, Vladimir Kempik > wrote: after reading manpage for getline I think I have found some elegant solution - to let getline manage size of buffer. Please review v2 of patch here http://cr.openjdk.java.net/~vkempik/8229872/webrev.02/ I?m still worried about syncing maxlinelen and linebuffer between Java_sun_nio_fs_LinuxNativeDispatcher_setmntent0 and Java_sun_nio_fs_LinuxNativeDispatcher_getmntent, if they are run from different threads. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Aug 20 20:16:41 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 20 Aug 2019 13:16:41 -0700 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> Message-ID: Hello, > On Aug 20, 2019, at 12:46 PM, Vladimir Kempik wrote: > > about L236, I really think I should not > > linebuffer will live throughtout app lifetime and will be realloced only when needed. It?s kind of cache now, for use by getmntent_r. > if I free linebuffer and set it to NULL, then I will have to set maxlinelen to 0 as well, otherwise getline will fail. > This means the app will have to recalc maxline ( and do some reallocs to linebuffer) every time setmntent0 is called. > > So I believe it?s better to let linebuffer leave, it?s not memory leak as we are tracking it and using it still. OK >>> On Aug 20, 2019, at 12:31 PM, Vladimir Kempik > wrote: >>> >>> after reading manpage for getline I think I have found some elegant solution - to let getline manage size of buffer. >>> >>> Please review v2 of patch here http://cr.openjdk.java.net/~vkempik/8229872/webrev.02/ >>> >>> I?m still worried about syncing maxlinelen and linebuffer between Java_sun_nio_fs_LinuxNativeDispatcher_setmntent0 and Java_sun_nio_fs_LinuxNativeDispatcher_getmntent, if they are run from different threads. Yes, I also wonder whether a synchronized(this) block or some such in LinuxFileSystem.getMountEntries(String) might be in order. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Aug 21 05:39:58 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 21 Aug 2019 06:39:58 +0100 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> Message-ID: <503abeab-0aae-937a-5c4b-b022e5f10dd1@oracle.com> On 20/08/2019 20:31, Vladimir Kempik wrote: > Hello > > after reading manpage for getline I think I have found some elegant solution - to let getline manage size of buffer. > > Please review v2 of patch here http://cr.openjdk.java.net/~vkempik/8229872/webrev.02/ > > I?m still worried about syncing maxlinelen and linebuffer between Java_sun_nio_fs_LinuxNativeDispatcher_setmntent0 and Java_sun_nio_fs_LinuxNativeDispatcher_getmntent, if they are run from different threads. > Finding the longest line to size the buffer seems a good approach to try. The way to do that is to add a getline and rewind to UnixNativeDispatcher and augment LinuxNativeDispatcher.getmntent to take a buffer size. Once you have those primitives then you can use them in LinuxFileStore.getMountEntries. -Alan From vtewar26 at in.ibm.com Wed Aug 21 05:49:04 2019 From: vtewar26 at in.ibm.com (Vyom Tewari26) Date: Wed, 21 Aug 2019 05:49:04 +0000 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: References: , <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> Message-ID: An HTML attachment was scrubbed... URL: From vtewar26 at in.ibm.com Wed Aug 21 06:01:33 2019 From: vtewar26 at in.ibm.com (Vyom Tewari26) Date: Wed, 21 Aug 2019 06:01:33 +0000 Subject: RFR 8211360 : Change #if DEF to #if defined(DEF) In-Reply-To: <0d38c07a-e697-49c6-dfc0-938a8ffa7820@oracle.com> References: <0d38c07a-e697-49c6-dfc0-938a8ffa7820@oracle.com> Message-ID: An HTML attachment was scrubbed... URL: From lance.andersen at oracle.com Wed Aug 21 14:46:02 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Wed, 21 Aug 2019 10:46:02 -0400 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: References: Message-ID: > On Aug 20, 2019, at 7:34 AM, Alan Bateman wrote: > > On 19/08/2019 21:48, Lance Andersen wrote: >> Hi >> >> The following patch for JDK-8229887 addresses an issue introduced by the performance fix in https://bugs.openjdk.java.net/browse/JDK-8222532 resulting in an error when trying to replace an entry in a ZIP/JAR file that was added with no-compression. >> >> The webrev can be found at http://cr.openjdk.java.net/~lancea/8229887/webrev.00/index.html > The update to ZipFileSystem looks good. Thank you Alan > > I think the test needs a bit of work and needs to test updating zip files where all entries are stored, all entries are deflated, and a mix of the two. I've created a test that does this if you need it. A revised webrev can be found at: http://cr.openjdk.java.net/~lancea/8229887/webrev.01/index.html This cleans up the existing test and adds some additional tests that were contributed by Alan. Mach5 tier1-3 are all clean Best Lance > > -Alan Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From ivan.gerasimov at oracle.com Wed Aug 21 20:58:17 2019 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Wed, 21 Aug 2019 13:58:17 -0700 Subject: RFR 8211360 : Change #if DEF to #if defined(DEF) In-Reply-To: References: <0d38c07a-e697-49c6-dfc0-938a8ffa7820@oracle.com> Message-ID: <308c2ef3-2b56-2541-40c9-d4ee9ad81cde@oracle.com> Thank you Brian and Vyom for reviewing it! With kind regards, Ivan On 8/20/19 11:01 PM, Vyom Tewari26 wrote: > Hi Ivan, > Changes look good to me. > Thanks, > Vyom > > ----- Original message ----- > From: Ivan Gerasimov > Sent by: "nio-dev" > To: core-libs-dev , > "nio-dev at openjdk.java.net" , > security-dev at openjdk.java.net > Cc: > Subject: [EXTERNAL] RFR 8211360 : Change #if DEF to #if defined(DEF) > Date: Tue, Aug 20, 2019 11:09 PM > Hello! > > It's a followup for JDK-8211146. > > With that fix several C-preprocessor statements of form #elif > __linux__ > were changed to more accurate #elif defined(__linux__). > > grep found a few more occurrences of the same pattern, which also > would > better be cleaned up. > > BUGURL: https://bugs.openjdk.java.net/browse/JDK-8211360 > WEBREV: http://cr.openjdk.java.net/~igerasim/8211360/00/webrev/ > > Mach5 control build/testing went fine on all platforms. > Would you please help review? > > -- > With kind regards, > Ivan Gerasimov > > -- With kind regards, Ivan Gerasimov -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Aug 22 09:42:55 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 22 Aug 2019 10:42:55 +0100 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: References: Message-ID: <7f0f5dd5-6793-7e24-f6b1-268b6598b76c@oracle.com> On 21/08/2019 15:46, Lance Andersen wrote: > > A revised webrev can be found at: > http://cr.openjdk.java.net/~lancea/8229887/webrev.01/index.html > > This cleans up the existing test and adds some additional tests that > were contributed by Alan. > Thanks for the update. Let's get this into jdk/jdk (for JDK 14) and let it bake. Separately we need to significantly expand the tests for this file system provider as there have been several issues recently that should have been caught by tests. -Alan From martinrb at google.com Thu Aug 22 15:11:17 2019 From: martinrb at google.com (Martin Buchholz) Date: Thu, 22 Aug 2019 08:11:17 -0700 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: <7f0f5dd5-6793-7e24-f6b1-268b6598b76c@oracle.com> References: <7f0f5dd5-6793-7e24-f6b1-268b6598b76c@oracle.com> Message-ID: I agree with Alan. The "env" that can be used to configure a ZipFileSystem appears to be completely undocumented, and some config parameters like zipinfo-time appear to be completely untested. Most ZipFileSystem tests should probably be run with both values of noCompression, including UpdateEntryTest. But let's not delay getting the bug fix into OpenJDK. On Thu, Aug 22, 2019 at 2:43 AM Alan Bateman wrote: > On 21/08/2019 15:46, Lance Andersen wrote: > > > > A revised webrev can be found at: > > http://cr.openjdk.java.net/~lancea/8229887/webrev.01/index.html > > > > This cleans up the existing test and adds some additional tests that > > were contributed by Alan. > > > Thanks for the update. Let's get this into jdk/jdk (for JDK 14) and let > it bake. Separately we need to significantly expand the tests for this > file system provider as there have been several issues recently that > should have been caught by tests. > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Aug 22 15:19:40 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 22 Aug 2019 16:19:40 +0100 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: References: <7f0f5dd5-6793-7e24-f6b1-268b6598b76c@oracle.com> Message-ID: On 22/08/2019 16:11, Martin Buchholz wrote: > I agree with Alan. > > The "env" that can be used to configure a ZipFileSystem appears to be > completely undocumented, and some config parameters like?zipinfo-time > appear to be completely untested. There's a table in the package description, it's been expanded quite a bit in the JDK 14 docs. -Alan. From martinrb at google.com Thu Aug 22 15:39:56 2019 From: martinrb at google.com (Martin Buchholz) Date: Thu, 22 Aug 2019 08:39:56 -0700 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: References: <7f0f5dd5-6793-7e24-f6b1-268b6598b76c@oracle.com> Message-ID: Thanks Alan. TIL that module-specific docs can be found in module-info.java https://download.java.net/java/early_access/jdk14/docs/api/jdk.zipfs/module-summary.html So indeed "env" is documented here, BUT the documentation is incomplete - at least noCompression and zipinfo-time are undocumented On Thu, Aug 22, 2019 at 8:19 AM Alan Bateman wrote: > On 22/08/2019 16:11, Martin Buchholz wrote: > > I agree with Alan. > > > > The "env" that can be used to configure a ZipFileSystem appears to be > > completely undocumented, and some config parameters like zipinfo-time > > appear to be completely untested. > > There's a table in the package description, it's been expanded quite a > bit in the JDK 14 docs. > > -Alan. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance.andersen at oracle.com Thu Aug 22 15:49:42 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Thu, 22 Aug 2019 11:49:42 -0400 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: References: <7f0f5dd5-6793-7e24-f6b1-268b6598b76c@oracle.com> Message-ID: <454DF2D7-6232-4149-9BC2-35483284B73D@oracle.com> Hi Martin, Yes there is additional work to do both WRT documentation and testing. We will continue to improve the documentation and the test coverage as we move forward. Best Lance > On Aug 22, 2019, at 11:39 AM, Martin Buchholz wrote: > > Thanks Alan. > > TIL that module-specific docs can be found in module-info.java > https://download.java.net/java/early_access/jdk14/docs/api/jdk.zipfs/module-summary.html > So indeed "env" is documented here, BUT the documentation is incomplete - at least noCompression and zipinfo-time are undocumented > > On Thu, Aug 22, 2019 at 8:19 AM Alan Bateman > wrote: > On 22/08/2019 16:11, Martin Buchholz wrote: > > I agree with Alan. > > > > The "env" that can be used to configure a ZipFileSystem appears to be > > completely undocumented, and some config parameters like zipinfo-time > > appear to be completely untested. > > There's a table in the package description, it's been expanded quite a > bit in the JDK 14 docs. > > -Alan. > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From vkempik at azul.com Fri Aug 23 14:41:30 2019 From: vkempik at azul.com (Vladimir Kempik) Date: Fri, 23 Aug 2019 14:41:30 +0000 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <503abeab-0aae-937a-5c4b-b022e5f10dd1@oracle.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> <503abeab-0aae-937a-5c4b-b022e5f10dd1@oracle.com> Message-ID: Hello please review v3 version of patch http://cr.openjdk.java.net/~vkempik/8229872/webrev.03/ no locks required now getline is used in reduced form, to only return linelen, nice comment about that in code included: +/* + getline wants malloc-ed buffer on input, which it might free. + Can't let getline to free java allocated array, + so can't pass anything on buffer input except NULL, + in this case getline will allocate the buffer and return it. +*/ Thanks, Vladimir > 21 ???. 2019 ?., ? 8:39, Alan Bateman ???????(?): > > On 20/08/2019 20:31, Vladimir Kempik wrote: >> Hello >> >> after reading manpage for getline I think I have found some elegant solution - to let getline manage size of buffer. >> >> Please review v2 of patch here http://cr.openjdk.java.net/~vkempik/8229872/webrev.02/ >> >> I?m still worried about syncing maxlinelen and linebuffer between Java_sun_nio_fs_LinuxNativeDispatcher_setmntent0 and Java_sun_nio_fs_LinuxNativeDispatcher_getmntent, if they are run from different threads. >> > Finding the longest line to size the buffer seems a good approach to try. The way to do that is to add a getline and rewind to UnixNativeDispatcher and augment LinuxNativeDispatcher.getmntent to take a buffer size. Once you have those primitives then you can use them in LinuxFileStore.getMountEntries. > > -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From Alan.Bateman at oracle.com Fri Aug 23 14:58:57 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 23 Aug 2019 15:58:57 +0100 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> <503abeab-0aae-937a-5c4b-b022e5f10dd1@oracle.com> Message-ID: On 23/08/2019 15:41, Vladimir Kempik wrote: > Hello > > please review v3 version of patch > > http://cr.openjdk.java.net/~vkempik/8229872/webrev.03/ > > no locks required now > getline is used in reduced form, to only return linelen, nice comment > about that in code included: I think the approach is reasonable but a few things need to be moved around to be consistent with the other code in this area. Can you move rewind to UnixNativeDispatcher so that it's with the other FILE* functions? I'm pretty sure you have to set errno to 0 before calling it and it will set it if there is an error. I think UnixNativeDispatcher should also define getline to wrap the libc function rather that getlinelen. I think we should change getmnent to take both a buffer and buffer length. If you look at the other wrapper methods you'll see how they use NativeBuffer to avoid the malloc in native code. -Alan From fweimer at redhat.com Fri Aug 23 15:06:46 2019 From: fweimer at redhat.com (Florian Weimer) Date: Fri, 23 Aug 2019 17:06:46 +0200 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: (Alan Bateman's message of "Fri, 23 Aug 2019 15:58:57 +0100") References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> <503abeab-0aae-937a-5c4b-b022e5f10dd1@oracle.com> Message-ID: <87a7bzsyyh.fsf@oldenburg2.str.redhat.com> * Alan Bateman: > Can you move rewind to UnixNativeDispatcher so that it's with the > other FILE* functions? I'm pretty sure you have to set errno to 0 > before calling it and it will set it if there is an error. You can check for errors using ferror. > I think UnixNativeDispatcher should also define getline to wrap the > libc function rather that getlinelen. That requires defining a new struct, for the internal state (buffer pointer and length). Thanks, Florian From vkempik at azul.com Fri Aug 23 15:18:49 2019 From: vkempik at azul.com (Vladimir Kempik) Date: Fri, 23 Aug 2019 15:18:49 +0000 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <87a7bzsyyh.fsf@oldenburg2.str.redhat.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> <503abeab-0aae-937a-5c4b-b022e5f10dd1@oracle.com> <87a7bzsyyh.fsf@oldenburg2.str.redhat.com> Message-ID: <0D734383-604F-414C-9F5D-27E5C617E04F@azul.com> Hello I?m not sure about full version of getline. getline requires malloced buffer and may update passed in buffer to a new size (via realloc) so it?s not safe to pass anything from java, (even NativeBuffer i think) to it. two ways to work it around: 1) minimal version of getline, like I did 2) in jni, allocate new, malloced buffer with same size as incoming buffer. pass it to getline. after getline finished, update incoming buffer with proper data (and increase it if needed). will still need malloc/free Alan, you really sure you want it to go second path while there is no consumer of full version of getline ? Thanks, Vladimir > 23 ???. 2019 ?., ? 18:06, Florian Weimer ???????(?): > > * Alan Bateman: > >> Can you move rewind to UnixNativeDispatcher so that it's with the >> other FILE* functions? I'm pretty sure you have to set errno to 0 >> before calling it and it will set it if there is an error. > > You can check for errors using ferror. > >> I think UnixNativeDispatcher should also define getline to wrap the >> libc function rather that getlinelen. > > That requires defining a new struct, for the internal state (buffer > pointer and length). > > Thanks, > Florian From Alan.Bateman at oracle.com Fri Aug 23 15:47:17 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 23 Aug 2019 16:47:17 +0100 Subject: RFR: 8229872: (fs) Increase buffer size used with getmntent In-Reply-To: <0D734383-604F-414C-9F5D-27E5C617E04F@azul.com> References: <1DF90E8C-07FB-4809-A2AD-04D9D0F2FF0D@azul.com> <87ftlxtllz.fsf@oldenburg2.str.redhat.com> <943b19d2-4846-3552-ab3d-6aaa598ece9e@oracle.com> <87v9uts4bz.fsf@oldenburg2.str.redhat.com> <2A80DD4F-6D09-477F-920C-B070601FBA54@azul.com> <87lfvps392.fsf@oldenburg2.str.redhat.com> <0FB89DBB-566C-4E32-8131-C15FA659022D@azul.com> <8736hvoqa3.fsf@oldenburg2.str.redhat.com> <20BDB142-8E77-4886-9902-CB1DE5CC4930@azul.com> <503abeab-0aae-937a-5c4b-b022e5f10dd1@oracle.com> <87a7bzsyyh.fsf@oldenburg2.str.redhat.com> <0D734383-604F-414C-9F5D-27E5C617E04F@azul.com> Message-ID: On 23/08/2019 16:18, Vladimir Kempik wrote: > Hello > > I?m not sure about full version of getline. > getline requires malloced buffer and may update passed in buffer to a new size (via realloc) so it?s not safe to pass anything from java, (even NativeBuffer i think) to it. > two ways to work it around: > 1) minimal version of getline, like I did > 2) in jni, allocate new, malloced buffer with same size as incoming buffer. pass it to getline. after getline finished, update incoming buffer with proper data (and increase it if needed). will still need malloc/free > > Alan, you really sure you want it to go second path while there is no consumer of full version of getline ? > I think we should introducing ad hoc native methods if possible. The native methods in this area are meant to be simple wrappers of syscall or libc function. There are few exceptions of course but the original intention was to be able to do as much in Java as possible. It should also make it easier to move this code to Project Panama in the future (there is already a version of UnixNativeDispatcher in that project). You are right that getline is a bit awkward because it is essentially returning both a pointer and a size. If we can't come up with something better then getlinelen might be okay but it needs cleanup and the comment changed to reflect what it does. -Alan From brian.burkhalter at oracle.com Fri Aug 23 18:33:21 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 23 Aug 2019 11:33:21 -0700 Subject: 8225461: (fs) Investigate filtering duplicates out of Iterable returned by java.nio.file.FileSystem::getFileStores In-Reply-To: <6B4F0C45-AE9B-476F-964D-E4423AAF9153@oracle.com> References: <9d73d427-f83a-cedc-45cb-1c6fa1299306@oracle.com> <13876EA1-D51B-4B67-ADED-C7174D46235F@oracle.com> <47D28DD6-B089-471E-A2D3-C13362ED8B3F@oracle.com> <6B4F0C45-AE9B-476F-964D-E4423AAF9153@oracle.com> Message-ID: <5409DB0A-7E6A-4707-9F0C-9677269ED0F4@oracle.com> Ping ? ! > On Aug 12, 2019, at 4:05 PM, Brian Burkhalter wrote: > >> On Aug 9, 2019, at 11:54 AM, Brian Burkhalter > wrote: >> >>> On Aug 9, 2019, at 11:49 AM, Bernd Eckenfels > wrote: >>> >>> I wonder if for the df case you should word-split the mountpoints, assuming that if duplicate lines are possible they might differ in any of the other columns which are not relevant for the test. (And i wonder if this as well as the /proc/mounts Support Blanks in the path) >> >> That might be a good idea. > > Here is a modified version [1] which uses only the device/filesystem and mount point fields of the df and /proc/mounts lines. > > Thanks, > > Brian > > [1] http://cr.openjdk.java.net/~bpb/8225461/webrev.02/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Sat Aug 24 08:31:52 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 24 Aug 2019 09:31:52 +0100 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: <454DF2D7-6232-4149-9BC2-35483284B73D@oracle.com> References: <7f0f5dd5-6793-7e24-f6b1-268b6598b76c@oracle.com> <454DF2D7-6232-4149-9BC2-35483284B73D@oracle.com> Message-ID: On 22/08/2019 16:49, Lance Andersen wrote: > Hi Martin, > > Yes there is additional work to do both WRT documentation and testing. > ?We will continue to improve the documentation ?and the test coverage > as we move forward. > Martin - are there folks in Google using zipfs in interesting ways that could be turned into tests? One long standing need, shared with the java.util.zip implementation, is automated interop testing with other zip tools where the spec might be interpreted differently (time stamps and zip64 are two areas where these type of issues can bite). -Alan From martinrb at google.com Sat Aug 24 22:54:59 2019 From: martinrb at google.com (Martin Buchholz) Date: Sat, 24 Aug 2019 15:54:59 -0700 Subject: RFR: 8229887: ZIP FS cannot replace an entry in a that was added via the STORED method In-Reply-To: References: <7f0f5dd5-6793-7e24-f6b1-268b6598b76c@oracle.com> <454DF2D7-6232-4149-9BC2-35483284B73D@oracle.com> Message-ID: On Sat, Aug 24, 2019 at 1:31 AM Alan Bateman wrote: > On 22/08/2019 16:49, Lance Andersen wrote: > > Hi Martin, > > > > Yes there is additional work to do both WRT documentation and testing. > > We will continue to improve the documentation and the test coverage > > as we move forward. > > > Martin - are there folks in Google using zipfs in interesting ways that > could be turned into tests? One long standing need, shared with the > java.util.zip implementation, is automated interop testing with other > zip tools where the spec might be interpreted differently (time stamps > and zip64 are two areas where these type of issues can bite). > Zip implementations in the JDK: ZipInputStream/ZipOutputStream ZipFile zipfs libjli/parse_manifest.c ... and probably more We don't encourage use of zipfs at Google, but people will use it when convenient. If all you want to do is replace a single entry in a zip file, it's hard to beat zipfs for convenience. It's hard work to write a program that makes small edits to a zip file while leaving it mostly intact. The recent undocumented switch of zipfs from STORED to DEFLATED is not nice to users. My test test/jdk/java/util/zip/EntryCount64k.java compares 3 different ways to read a zip file, but zipfs is not one of them :-( . I'd also like it to try unzip or zipinfo if they appear to be installed on the system. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Aug 26 18:45:21 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 26 Aug 2019 11:45:21 -0700 Subject: 8229280: Test failures on several linux hosts after JDK-8181493 In-Reply-To: <8DE2D4E9-E312-4E73-A692-DE35141BFC2A@oracle.com> References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> <7b4c2bd4-4e3d-70f0-9d8b-b0d24ae038a0@oracle.com> <8DE2D4E9-E312-4E73-A692-DE35141BFC2A@oracle.com> Message-ID: <638E1D14-E6F7-4BD1-97F2-1579F437D930@oracle.com> Ping ... > On Aug 9, 2019, at 5:05 PM, Brian Burkhalter wrote: > > Here is a potential fix [1] for the issue [2] provoked by the fix for [3]. The change is not to require strict equality in timestamps except for some file system types which are known to support nanosecond precision and are explicitly identified. For non-strict timestamp comparisons, equality is defined by the timestamp read back from the file having a number of trailing zeros which is a multiple of 3, and having a prefix which is equal to a prefix of the same length of the nanosecond precision timestamp. > > If there are more file system types which should be added to the list of strictly tested types, please let me know. > > Thanks, > > Brian > > [1] http://cr.openjdk.java.net/~bpb/8229280/webrev.00/ > [2] https://bugs.openjdk.java.net/browse/JDK-8229280 > [3] https://bugs.openjdk.java.net/browse/JDK-8181493 > > >> On Aug 8, 2019, at 8:52 AM, Brian Burkhalter > wrote: >> >> HI Christoph, >> >> I don?t know of a programmatic way to check whether nanosecond precision is supported. Do you happen to know the file system type (and os.name property)? Right now the test has this exclusion >> 53 if (System.getProperty("os.name").toLowerCase().startsWith("mac") && >> 54 store.name().equalsIgnoreCase("hfs")) { >> 55 System.err.println >> 56 ("HFS on macOS does not have nsec timestamps: skipping test"); >> 57 return; >> 58 } >> Thanks, >> >> Brian >> >>> On Aug 8, 2019, at 7:56 AM, Langer, Christoph > wrote: >>> >>> it seems that we?re encountering this behavior on old Suse SLES 11.3 and 11.4 machines, both ppc64 and linuxx86_64. >>> >>> Maybe these can somehow be excluded from the test? Or the test can detect upfront when nanotime isn?t supported? >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Aug 28 09:16:12 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 28 Aug 2019 10:16:12 +0100 Subject: 8229280: Test failures on several linux hosts after JDK-8181493 In-Reply-To: <638E1D14-E6F7-4BD1-97F2-1579F437D930@oracle.com> References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> <7b4c2bd4-4e3d-70f0-9d8b-b0d24ae038a0@oracle.com> <8DE2D4E9-E312-4E73-A692-DE35141BFC2A@oracle.com> <638E1D14-E6F7-4BD1-97F2-1579F437D930@oracle.com> Message-ID: On 26/08/2019 19:45, Brian Burkhalter wrote: > Ping ... Can this test be changed to call UnixNativeDispatcher.futimensSupported reflectively? I think that would eliminate the most of the complexity and retry in this test. -Alan From Alan.Bateman at oracle.com Wed Aug 28 09:26:00 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 28 Aug 2019 10:26:00 +0100 Subject: 8225461: (fs) Investigate filtering duplicates out of Iterable returned by java.nio.file.FileSystem::getFileStores In-Reply-To: <5409DB0A-7E6A-4707-9F0C-9677269ED0F4@oracle.com> References: <9d73d427-f83a-cedc-45cb-1c6fa1299306@oracle.com> <13876EA1-D51B-4B67-ADED-C7174D46235F@oracle.com> <47D28DD6-B089-471E-A2D3-C13362ED8B3F@oracle.com> <6B4F0C45-AE9B-476F-964D-E4423AAF9153@oracle.com> <5409DB0A-7E6A-4707-9F0C-9677269ED0F4@oracle.com> Message-ID: On 23/08/2019 19:33, Brian Burkhalter wrote: > Ping ? ! As a short term workaround in the test then this is probably okay although "replicates" looks a bit odd. I would much prefer if we find time to get to root of these issues so that the implementation can handle these cases where there are duplicate or overlay mounts. I realize this will require work to setup these scenarios but we have to get to the underlying issue to avoid the continuous trickle of bug reports with unusual configurations. -Alan From brian.burkhalter at oracle.com Thu Aug 29 00:56:24 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 28 Aug 2019 17:56:24 -0700 Subject: 8229280: Test failures on several linux hosts after JDK-8181493 In-Reply-To: References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> <7b4c2bd4-4e3d-70f0-9d8b-b0d24ae038a0@oracle.com> <8DE2D4E9-E312-4E73-A692-DE35141BFC2A@oracle.com> <638E1D14-E6F7-4BD1-97F2-1579F437D930@oracle.com> Message-ID: > On Aug 28, 2019, at 2:16 AM, Alan Bateman wrote: > > On 26/08/2019 19:45, Brian Burkhalter wrote: >> Ping ... > Can this test be changed to call UnixNativeDispatcher.futimensSupported reflectively? I think that would eliminate the most of the complexity and retry in this test. Done [1]. Note however that futimens() can be supported without it actually being able to set the timestamps to nanosecond precision, e.g., for HFS file systems on macOS. Therefore the test will still pass if the actual and expected values are not equal due to truncation of a multiple of three digits of precision. Thanks, Brian [1] http://cr.openjdk.java.net/~bpb/8229280/webrev.01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Aug 30 23:59:00 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 30 Aug 2019 16:59:00 -0700 Subject: 8225461: (fs) Investigate filtering duplicates out of Iterable returned by java.nio.file.FileSystem::getFileStores In-Reply-To: References: <9d73d427-f83a-cedc-45cb-1c6fa1299306@oracle.com> <13876EA1-D51B-4B67-ADED-C7174D46235F@oracle.com> <47D28DD6-B089-471E-A2D3-C13362ED8B3F@oracle.com> <6B4F0C45-AE9B-476F-964D-E4423AAF9153@oracle.com> <5409DB0A-7E6A-4707-9F0C-9677269ED0F4@oracle.com> Message-ID: <9F2B2DA6-6351-4465-851A-05D14AA8382A@oracle.com> > On Aug 28, 2019, at 2:26 AM, Alan Bateman wrote: > > On 23/08/2019 19:33, Brian Burkhalter wrote: >> Ping ? ! > As a short term workaround in the test then this is probably okay although "replicates" looks a bit odd. I would much prefer if we find time to get to root of these issues so that the implementation can handle these cases where there are duplicate or overlay mounts. I realize this will require work to setup these scenarios but we have to get to the underlying issue to avoid the continuous trickle of bug reports with unusual configurations. This particular patch was really intended to avoid spurious failures due to misconfigured systems in the CI testing environment. As such it might instead be better addressed under the issue [1]. The issue [2] of the current discussion thread could then be left open to address handling these weird cases in the implementation itself. Thanks, Brian [1] https://bugs.openjdk.java.net/browse/JDK-8225471 [2] https://bugs.openjdk.java.net/browse/JDK-8225461 -------------- next part -------------- An HTML attachment was scrubbed... URL: